# cloud
[](https://pypi.org/project/great-expectations_cloud/#history)
[](https://hub.docker.com/r/greatexpectations/agent)
[](https://github.com/great-expectations/cloud/actions/workflows/ci.yaml)
[](https://results.pre-commit.ci/latest/github/great-expectations/cloud/main)
[](https://codecov.io/gh/great-expectations/cloud)
[](https://github.com/astral-sh/ruff)
## Quick Start
### Python
#### Install
```console
pip install great_expectations_cloud
```
##### Optional Dependencies
```console
pip install 'great_expectations_cloud[sql]'
```
```console
$ gx-agent --help
usage: gx-agent [-h] [--log-level LOG_LEVEL] [--skip-log-file SKIP_LOG_FILE] [--log-cfg-file LOG_CFG_FILE] [--version]
optional arguments:
-h, --help show this help message and exit
--log-level LOG_LEVEL
Level of logging to use. Defaults to WARNING.
--skip-log-file SKIP_LOG_FILE
Skip writing debug logs to a file. Defaults to False. Does not affect logging to stdout/stderr.
--log-cfg-file LOG_CFG_FILE
Path to a logging configuration json file. Supersedes --log-level and --skip-log-file.
--version Show the GX Agent version.
```
#### Set env variables
`GX_CLOUD_ACCESS_TOKEN`
`GX_CLOUD_ORGANIZATION_ID`
### Start the Agent
If you intend to run the Agent against local services (Cloud backend or datasources) run the Agent outside of the container.
```
gx-agent
```
### Docker
[Building and running the Agent with Docker](#building-and-running-the-gx-agent-image)
## Dev Setup
See also [CONTRIBUTING.md](https://github.com/great-expectations/cloud/blob/main/CONTRIBUTING.md)
1. [Install `poetry`](https://python-poetry.org/docs/#installation)
- [`pipx install poetry`](https://python-poetry.org/docs/#installing-with-pipx)
2. Set up virtual environment and install dependencies
- `poetry install --sync`
3. Activate your virtual environment
- `poetry shell`
4. Set up precommit hooks
- `pre-commit install`
### Developer Tasks
Common developer tasks are available via `invoke` (defined in `tasks.py`)
`invoke --list` to see available tasks.
#### Synchronize Dependencies
To ensure you are using the latest version of the core and development dependencies run `poetry install --sync`.
Also available as an invoke task.
```console
invoke deps
```
#### Updating `poetry.lock` dependencies
The dependencies installed in our CI and the Docker build step are determined by the [poetry.lock file](https://python-poetry.org/docs/basic-usage/#installing-with-poetrylock).
[To update only a specific dependency](https://python-poetry.org/docs/cli/#update) (such as `great_expectations`) ...
```console
poetry update great_expectations
```
[To resolve and update all dependencies ...](https://python-poetry.org/docs/cli/#lock)
```console
poetry lock
```
In either case, the updated `poetry.lock` file must be committed and merged to main.
#### Building and Running the GX Agent Image
To build the GX Agent Docker image, run the following in the root dir:
```
invoke docker
```
Running the GX Agent:
```
invoke docker --run
```
or
```
docker run --env GX_CLOUD_ACCESS_TOKEN="<GX_TOKEN>" --env GX_CLOUD_ORGANIZATION_ID="<GX_ORG_ID>" gx/agent
```
Now go into GX Cloud and issue commands for the GX Agent to run, such as generating an Expectation Suite for a Data Source.
> Note if you are pushing out a new image update the image tag version in `containerize-agent.yaml`. The image will be built and pushed out via GitHub Actions.
#### Example Data
The contents from [/examples/agent/data](https://github.com/great-expectations/cloud/tree/main/examples/agent/data) will be copied to `/data` for the Docker container.
#### Adding an action to the Agent
1. Make a new action in `great_expectations_cloud/agent/actions/` in a separate file.
2. Register your action in the file it was created in using `great_expectations_cloud.agent.event_handler.register_event_action()`. Register for the major version of GX Core that the action applies to, e.g. `register_event_action("1", RunCheckpointEvent, RunCheckpointAction)` registers the action for major version 1 of GX Core (e.g. 1.0.0).
3. Import your action in `great_expectations_cloud/agent/actions/__init__.py`
Note: The Agent is core-version specific but this registration mechanism allows us to preemptively work on actions for future versions of GX Core while still supporting the existing latest major version.
### Release Process
#### Versioning
This is the version that will be used for the Docker image tag as well.
_Standard Release_:
The versioning scheme is `YYYYMMDD.{release_number}` where:
- the date is the date of the release
- the release number starts at 0 for the first release of the day
- the release number is incremented for each release within the same day
For example: `20240402.0`
_Pre-release_:
The versioning scheme is `YYYYMMDD.{release_number}.dev{dev_number}`
- the date is the date of the release
- the dev number starts at 0 for the first pre-release of the day
- the dev number is incremented for each pre-release within the same day
- the release number is the release that this pre-release is for
For example: `20240403.0.dev0` is the first pre-release for the `20240403.0` release.
For example, imagine the following sequence of releases given for a day with two releases:
- `20240403.0.dev0`
- `20240403.0.dev1`
- `20240403.0`
- `20240403.1.dev0`
- `20240403.1`
There can be days with no standard releases, only pre-releases or days with no pre-release or standard release at all.
#### Pre-releases
Pre-releases are completed automatically with each merge to the `main` branch.
The version is updated in `pyproject.toml` and a pre-release is created on PyPi.
A new Docker tag will also be generated and pushed to [Docker Hub](https://hub.docker.com/r/greatexpectations/agent)
**Manual Pre-releases**
NOTE: CI will automatically create pre-releases on merges to `main`. Instead of manually creating pre-releases, consider using the CI process. This is only for exceptional cases.
To manually create a pre-release, run the following command to update the version in `pyproject.toml` and then merge it to `main` in a standalone PR:
```console
invoke pre-release
```
This will create a new pre-release version. On the next merge to `main`, the release will be uploaded to PyPi.
A new Docker tag will also be generated and pushed to [Docker Hub](https://hub.docker.com/r/greatexpectations/agent)
#### Releases
Releases will be completed on a regular basis by the maintainers of the project and with any release of [GX Core](https://github.com/great-expectations/great_expectations)
For maintainers, to create a release, run the following command to update the version in `pyproject.toml` and then
merge it to `main` in a standalone PR:
```console
invoke release
```
This will create a new release version. On the next merge to `main`, the release will be uploaded to PyPi.
A new Docker tag will also be generated and pushed to [Docker Hub](https://hub.docker.com/r/greatexpectations/agent). In addition, releases will be tagged with `stable` and `latest` tags.
#### GitHub Workflow for releasing
We use the GitHub Actions workflow to automate the release and pre-release process. There are two workflows involved:
1. [CI](https://github.com/great-expectations/cloud/blob/main/.github/workflows/ci.yaml) - This workflow runs on each pull request and will update the version in `pyproject.toml` to the pre-release version if the version is not already manually updated in the PR. It will also run the tests and linting.
2. [Containerize Agent](https://github.com/great-expectations/cloud/blob/main/.github/workflows/containerize-agent.yaml) - This workflows runs on merge with `main` and will create a new Docker image and push it to Docker Hub and PyPi. It uses the version in `pyproject.toml`.
A visual representation of the workflow is shown [here](https://github.com/great-expectations/cloud/blob/main/.github/workflows/agent_release_workflows.png)
### Dependabot and Releases/Pre-releases
GitHub's Dependabot regularly checks our dependencies for vulnerabilty-based updates and proposes PRs to update dependency version numbers accordingly.
Dependabot may only update the `poetry.lock` file. If only changes to `poetry.lock` are made, this may be done in a pre-release.
For changes to the `pyproject.toml` file:
- If the version of a tool in the `[tool.poetry.group.dev.dependencies]` group is updated, this may be done without any version bump.
- While doing this, make sure any version references in the pre-commit config `.pre-commit-config.yaml` are kept in sync (e.g., ruff).
- For other dependency updates or package build metadata changes, a new release should be orchestrated. This includes updates in the following sections:
- `[tool.poetry.dependencies]`
- `[tool.poetry.group.*.dependencies]` where `*` is the name of the group (not including the `dev` group)
- To stop the auto-version bump add the `no version bump` label to the PR. Use this when:
- Only modifying dev dependencies.
- Only modifying tests that do not change functionality.
Raw data
{
"_id": null,
"home_page": "https://greatexpectations.io",
"name": "great-expectations-cloud",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.12,>=3.8",
"maintainer_email": null,
"keywords": null,
"author": "The Great Expectations Team",
"author_email": "team@greatexpectations.io",
"download_url": "https://files.pythonhosted.org/packages/9a/e7/a255a023eef375986f206b3261b0a961b102108acbac39f4f3d0ecd81e35/great_expectations_cloud-20240521.0.tar.gz",
"platform": null,
"description": "# cloud\n\n[](https://pypi.org/project/great-expectations_cloud/#history)\n[](https://hub.docker.com/r/greatexpectations/agent)\n[](https://github.com/great-expectations/cloud/actions/workflows/ci.yaml)\n[](https://results.pre-commit.ci/latest/github/great-expectations/cloud/main)\n[](https://codecov.io/gh/great-expectations/cloud)\n[](https://github.com/astral-sh/ruff)\n\n## Quick Start\n\n### Python\n\n#### Install\n\n```console\npip install great_expectations_cloud\n```\n\n##### Optional Dependencies\n\n```console\npip install 'great_expectations_cloud[sql]'\n```\n\n```console\n$ gx-agent --help\nusage: gx-agent [-h] [--log-level LOG_LEVEL] [--skip-log-file SKIP_LOG_FILE] [--log-cfg-file LOG_CFG_FILE] [--version]\n\noptional arguments:\n -h, --help show this help message and exit\n --log-level LOG_LEVEL\n Level of logging to use. Defaults to WARNING.\n --skip-log-file SKIP_LOG_FILE\n Skip writing debug logs to a file. Defaults to False. Does not affect logging to stdout/stderr.\n --log-cfg-file LOG_CFG_FILE\n Path to a logging configuration json file. Supersedes --log-level and --skip-log-file.\n --version Show the GX Agent version.\n```\n\n#### Set env variables\n\n`GX_CLOUD_ACCESS_TOKEN`\n`GX_CLOUD_ORGANIZATION_ID`\n\n### Start the Agent\n\nIf you intend to run the Agent against local services (Cloud backend or datasources) run the Agent outside of the container.\n\n```\ngx-agent\n```\n\n### Docker\n\n[Building and running the Agent with Docker](#building-and-running-the-gx-agent-image)\n\n## Dev Setup\n\nSee also [CONTRIBUTING.md](https://github.com/great-expectations/cloud/blob/main/CONTRIBUTING.md)\n\n1. [Install `poetry`](https://python-poetry.org/docs/#installation)\n - [`pipx install poetry`](https://python-poetry.org/docs/#installing-with-pipx)\n2. Set up virtual environment and install dependencies\n - `poetry install --sync`\n3. Activate your virtual environment\n - `poetry shell`\n4. Set up precommit hooks\n - `pre-commit install`\n\n### Developer Tasks\n\nCommon developer tasks are available via `invoke` (defined in `tasks.py`)\n\n`invoke --list` to see available tasks.\n\n#### Synchronize Dependencies\n\nTo ensure you are using the latest version of the core and development dependencies run `poetry install --sync`.\nAlso available as an invoke task.\n\n```console\ninvoke deps\n```\n\n#### Updating `poetry.lock` dependencies\n\nThe dependencies installed in our CI and the Docker build step are determined by the [poetry.lock file](https://python-poetry.org/docs/basic-usage/#installing-with-poetrylock).\n\n[To update only a specific dependency](https://python-poetry.org/docs/cli/#update) (such as `great_expectations`) ...\n\n```console\npoetry update great_expectations\n```\n\n[To resolve and update all dependencies ...](https://python-poetry.org/docs/cli/#lock)\n\n```console\npoetry lock\n```\n\nIn either case, the updated `poetry.lock` file must be committed and merged to main.\n\n#### Building and Running the GX Agent Image\n\nTo build the GX Agent Docker image, run the following in the root dir:\n\n```\ninvoke docker\n```\n\nRunning the GX Agent:\n\n```\ninvoke docker --run\n```\n\nor\n\n```\ndocker run --env GX_CLOUD_ACCESS_TOKEN=\"<GX_TOKEN>\" --env GX_CLOUD_ORGANIZATION_ID=\"<GX_ORG_ID>\" gx/agent\n```\n\nNow go into GX Cloud and issue commands for the GX Agent to run, such as generating an Expectation Suite for a Data Source.\n\n> Note if you are pushing out a new image update the image tag version in `containerize-agent.yaml`. The image will be built and pushed out via GitHub Actions.\n\n#### Example Data\n\nThe contents from [/examples/agent/data](https://github.com/great-expectations/cloud/tree/main/examples/agent/data) will be copied to `/data` for the Docker container.\n\n#### Adding an action to the Agent\n\n1. Make a new action in `great_expectations_cloud/agent/actions/` in a separate file.\n2. Register your action in the file it was created in using `great_expectations_cloud.agent.event_handler.register_event_action()`. Register for the major version of GX Core that the action applies to, e.g. `register_event_action(\"1\", RunCheckpointEvent, RunCheckpointAction)` registers the action for major version 1 of GX Core (e.g. 1.0.0).\n3. Import your action in `great_expectations_cloud/agent/actions/__init__.py`\n\nNote: The Agent is core-version specific but this registration mechanism allows us to preemptively work on actions for future versions of GX Core while still supporting the existing latest major version.\n\n### Release Process\n\n#### Versioning\n\nThis is the version that will be used for the Docker image tag as well.\n\n_Standard Release_:\nThe versioning scheme is `YYYYMMDD.{release_number}` where:\n\n- the date is the date of the release\n- the release number starts at 0 for the first release of the day\n- the release number is incremented for each release within the same day\n\nFor example: `20240402.0`\n\n_Pre-release_:\nThe versioning scheme is `YYYYMMDD.{release_number}.dev{dev_number}`\n\n- the date is the date of the release\n- the dev number starts at 0 for the first pre-release of the day\n- the dev number is incremented for each pre-release within the same day\n- the release number is the release that this pre-release is for\n\nFor example: `20240403.0.dev0` is the first pre-release for the `20240403.0` release.\n\nFor example, imagine the following sequence of releases given for a day with two releases:\n\n- `20240403.0.dev0`\n- `20240403.0.dev1`\n- `20240403.0`\n- `20240403.1.dev0`\n- `20240403.1`\n\nThere can be days with no standard releases, only pre-releases or days with no pre-release or standard release at all.\n\n#### Pre-releases\n\nPre-releases are completed automatically with each merge to the `main` branch.\nThe version is updated in `pyproject.toml` and a pre-release is created on PyPi.\nA new Docker tag will also be generated and pushed to [Docker Hub](https://hub.docker.com/r/greatexpectations/agent)\n\n**Manual Pre-releases**\n\nNOTE: CI will automatically create pre-releases on merges to `main`. Instead of manually creating pre-releases, consider using the CI process. This is only for exceptional cases.\n\nTo manually create a pre-release, run the following command to update the version in `pyproject.toml` and then merge it to `main` in a standalone PR:\n\n```console\ninvoke pre-release\n```\n\nThis will create a new pre-release version. On the next merge to `main`, the release will be uploaded to PyPi.\nA new Docker tag will also be generated and pushed to [Docker Hub](https://hub.docker.com/r/greatexpectations/agent)\n\n#### Releases\n\nReleases will be completed on a regular basis by the maintainers of the project and with any release of [GX Core](https://github.com/great-expectations/great_expectations)\n\nFor maintainers, to create a release, run the following command to update the version in `pyproject.toml` and then\nmerge it to `main` in a standalone PR:\n\n```console\ninvoke release\n```\n\nThis will create a new release version. On the next merge to `main`, the release will be uploaded to PyPi.\nA new Docker tag will also be generated and pushed to [Docker Hub](https://hub.docker.com/r/greatexpectations/agent). In addition, releases will be tagged with `stable` and `latest` tags.\n\n#### GitHub Workflow for releasing\n\nWe use the GitHub Actions workflow to automate the release and pre-release process. There are two workflows involved:\n\n1. [CI](https://github.com/great-expectations/cloud/blob/main/.github/workflows/ci.yaml) - This workflow runs on each pull request and will update the version in `pyproject.toml` to the pre-release version if the version is not already manually updated in the PR. It will also run the tests and linting.\n\n2. [Containerize Agent](https://github.com/great-expectations/cloud/blob/main/.github/workflows/containerize-agent.yaml) - This workflows runs on merge with `main` and will create a new Docker image and push it to Docker Hub and PyPi. It uses the version in `pyproject.toml`.\n\nA visual representation of the workflow is shown [here](https://github.com/great-expectations/cloud/blob/main/.github/workflows/agent_release_workflows.png)\n\n### Dependabot and Releases/Pre-releases\nGitHub's Dependabot regularly checks our dependencies for vulnerabilty-based updates and proposes PRs to update dependency version numbers accordingly.\n\nDependabot may only update the `poetry.lock` file. If only changes to `poetry.lock` are made, this may be done in a pre-release.\n\nFor changes to the `pyproject.toml` file:\n- If the version of a tool in the `[tool.poetry.group.dev.dependencies]` group is updated, this may be done without any version bump.\n - While doing this, make sure any version references in the pre-commit config `.pre-commit-config.yaml` are kept in sync (e.g., ruff).\n- For other dependency updates or package build metadata changes, a new release should be orchestrated. This includes updates in the following sections:\n - `[tool.poetry.dependencies]`\n - `[tool.poetry.group.*.dependencies]` where `*` is the name of the group (not including the `dev` group)\n- To stop the auto-version bump add the `no version bump` label to the PR. Use this when:\n - Only modifying dev dependencies.\n - Only modifying tests that do not change functionality.\n\n",
"bugtrack_url": null,
"license": "Proprietary",
"summary": "Great Expectations Cloud",
"version": "20240521.0",
"project_urls": {
"Homepage": "https://greatexpectations.io",
"Repository": "https://github.com/great-expectations/cloud"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "13f432a550ab5b6ea66169a7b1db305d591afb34205c403b048c10d5fe2a73a6",
"md5": "db6d5e92d450bb49fadd133569b4ac7c",
"sha256": "f564391c5b64068969e73765734a702250f299d7575e5764217af23729103007"
},
"downloads": -1,
"filename": "great_expectations_cloud-20240521.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "db6d5e92d450bb49fadd133569b4ac7c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.8",
"size": 37470,
"upload_time": "2024-05-21T21:21:46",
"upload_time_iso_8601": "2024-05-21T21:21:46.505421Z",
"url": "https://files.pythonhosted.org/packages/13/f4/32a550ab5b6ea66169a7b1db305d591afb34205c403b048c10d5fe2a73a6/great_expectations_cloud-20240521.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9ae7a255a023eef375986f206b3261b0a961b102108acbac39f4f3d0ecd81e35",
"md5": "bca6e961836a04a4296112cf618abaac",
"sha256": "501a2ce6a712adb3511162384f2b62528366ec216631697ce940ca6c7f08bfaf"
},
"downloads": -1,
"filename": "great_expectations_cloud-20240521.0.tar.gz",
"has_sig": false,
"md5_digest": "bca6e961836a04a4296112cf618abaac",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.8",
"size": 31049,
"upload_time": "2024-05-21T21:21:48",
"upload_time_iso_8601": "2024-05-21T21:21:48.826684Z",
"url": "https://files.pythonhosted.org/packages/9a/e7/a255a023eef375986f206b3261b0a961b102108acbac39f4f3d0ecd81e35/great_expectations_cloud-20240521.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-21 21:21:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "great-expectations",
"github_project": "cloud",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "great-expectations-cloud"
}
JSON
