# PDS Registry Client
This is a prototype implementation of a request-signing utility for use with serverless OpenSearch (AOSS). It is
(currently) intended to provide a curl-like interface for querying PDS Registry's AOSS instance using a Cognito user
identity.
Additional functionality may be built out in the future.
## Prerequisites
- Personal user/pass credentials for a Cognito user authorized for Registry AOSS
- Python >=3.9
- Environment variables (contact developer for values)
```bash
export REQUEST_SIGNER_AWS_ACCOUNT=''
export REQUEST_SIGNER_AWS_REGION=''
export REQUEST_SIGNER_CLIENT_ID=''
export REQUEST_SIGNER_USER_POOL_ID=''
export REQUEST_SIGNER_IDENTITY_POOL_ID=''
export REQUEST_SIGNER_AOSS_ENDPOINT=''
export REQUEST_SIGNER_COGNITO_USER=''
export REQUEST_SIGNER_COGNITO_PASSWORD=''
```
## Developer Quickstart
1. Clone the repository
```
git clone https://github.com/NASA-PDS/registry-client.git
cd registry-client
```
2. Create a virtual environment
```
python -m venv venv
source ./venv/bin/activate
```
3. Install the tool to the virtual environment
```
pip install --editable .[dev]
```
4. Run the tool directly
```
registry-client --help
```
## Code of Conduct
All users and developers of the NASA-PDS software are expected to abide by our [Code of Conduct](https://github.com/NASA-PDS/.github/blob/main/CODE_OF_CONDUCT.md). Please read this to ensure you understand the expectations of our community.
## Development
To develop this project, use your favorite text editor, or an integrated development environment with Python support, such as [PyCharm](https://www.jetbrains.com/pycharm/).
### Contributing
For information on how to contribute to NASA-PDS codebases please take a look at our [Contributing guidelines](https://github.com/NASA-PDS/.github/blob/main/CONTRIBUTING.md).
### Installation
Install in editable mode and with extra developer dependencies into your virtual environment of choice:
pip install --editable '.[dev]'
Make a baseline for any secrets (email addresses, passwords, API keys, etc.) in the repository:
detect-secrets scan . \
--all-files \
--disable-plugin AbsolutePathDetectorExperimental \
--exclude-files '\.secrets..*' \
--exclude-files '\.git.*' \
--exclude-files '\.mypy_cache' \
--exclude-files '\.pytest_cache' \
--exclude-files '\.tox' \
--exclude-files '\.venv' \
--exclude-files 'venv' \
--exclude-files 'dist' \
--exclude-files 'build' \
--exclude-files '.*\.egg-info' > .secrets.baseline
Review the secrets to determine which should be allowed and which are false positives:
detect-secrets audit .secrets.baseline
Please remove any secrets that should not be seen by the public. You can then add the baseline file to the commit:
git add .secrets.baseline
Then, configure the `pre-commit` hooks:
pre-commit install
pre-commit install -t pre-push
pre-commit install -t prepare-commit-msg
pre-commit install -t commit-msg
These hooks then will check for any future commits that might contain secrets. They also check code formatting, PEP8 compliance, type hints, etc.
👉 **Note:** A one time setup is required both to support `detect-secrets` and in your global Git configuration. See [the wiki entry on Secrets](https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Git-and-Github-Guide#detect-secrets) to learn how.
### Packaging
To isolate and be able to re-produce the environment for this package, you should use a [Python Virtual Environment](https://docs.python.org/3/tutorial/venv.html). To do so, run:
python -m venv venv
Then exclusively use `venv/bin/python`, `venv/bin/pip`, etc.
If you have `tox` installed and would like it to create your environment and install dependencies for you run:
tox --devenv <name you'd like for env> -e dev
Dependencies for development are specified as the `dev` `extras_require` in `setup.cfg`; they are installed into the virtual environment as follows:
pip install --editable '.[dev]'
All the source code is in a sub-directory under `src`.
You should update the `setup.cfg` file with:
- name of your module
- license, default apache, update if needed
- description
- download url, when you release your package on github add the url here
- keywords
- classifiers
- install_requires, add the dependencies of you package
- extras_require, add the development Dependencies of your package
- entry_points, when your package can be called in command line, this helps to deploy command lines entry points pointing to scripts in your package
For the packaging details, see https://packaging.python.org/tutorials/packaging-projects/ as a reference.
### Configuration
It is convenient to use ConfigParser package to manage configuration. It allows a default configuration which can be overwritten by the user in a specific file in their environment. See https://pymotw.com/2/ConfigParser/
For example:
candidates = ['my_pds_module.ini', 'my_pds_module.ini.default']
found = parser.read(candidates)
### Logs
You should not use `print()`vin the purpose of logging information on the execution of your code. Depending on where the code runs these information could be redirected to specific log files.
To make that work, start each Python file with:
```python
"""My module."""
import logging
logger = logging.getLogger(__name__)
```
To log a message:
logger.info("my message")
In your `main` routine, include:
logging.basicConfig(level=logging.INFO)
to get a basic logging system configured.
### Tooling
The `dev` `extras_require` included in the template repo installs `black`, `flake8` (plus some plugins), and `mypy` along with default configuration for all of them. You can run all of these (and more!) with:
tox -e lint
### Code Style
So that your code is readable, you should comply with the [PEP8 style guide](https://www.python.org/dev/peps/pep-0008/). Our code style is automatically enforced in via [black](https://pypi.org/project/black/) and [flake8](https://flake8.pycqa.org/en/latest/). See the [Tooling section](#-tooling) for information on invoking the linting pipeline.
âť—Important note for template usersâť—
The included [pre-commit configuration file](.pre-commit-config.yaml) executes `flake8` (along with `mypy`) across the entire `src` folder and not only on changed files. If you're converting a pre-existing code base over to this template that may result in a lot of errors that you aren't ready to deal with.
You can instead execute `flake8` only over a diff of the current changes being made by modifying the `pre-commit` `entry` line:
entry: git diff -u | flake8 --diff
Or you can change the `pre-commit` config so `flake8` is only called on changed files which match a certain filtering criteria:
- repo: local
hooks:
- id: flake8
name: flake8
entry: flake8
files: ^src/|tests/
language: system
### Recommended Libraries
Python offers a large variety of libraries. In PDS scope, for the most current usage we should use:
| Library | Usage |
|--------------|------------------------------------------------ |
| configparser | manage and parse configuration files |
| argparse | command line argument documentation and parsing |
| requests | interact with web APIs |
| lxml | read/write XML files |
| json | read/write JSON files |
| pyyaml | read/write YAML files |
| pystache | generate files from templates |
Some of these are built into Python 3; others are open source add-ons you can include in your `requirements.txt`.
### Tests
This section describes testing for your package.
A complete "build" including test execution, linting (`mypy`, `black`, `flake8`, etc.), and documentation build is executed via:
tox
#### Unit tests
Your project should have built-in unit tests, functional, validation, acceptance, etc., tests.
For unit testing, check out the [unittest](https://docs.python.org/3/library/unittest.html) module, built into Python 3.
Tests objects should be in packages `test` modules or preferably in project 'tests' directory which mirrors the project package structure.
Our unit tests are launched with command:
pytest
If you want your tests to run automatically as you make changes start up `pytest` in watch mode with:
ptw
#### Integration/Behavioral Tests
One should use the `behave package` and push the test results to "testrail".
See an example in https://github.com/NASA-PDS/pds-doi-service#behavioral-testing-for-integration--testing
### Documentation
Your project should use [Sphinx](https://www.sphinx-doc.org/en/master/) to build its documentation. PDS' documentation template is already configured as part of the default build. You can build your projects docs with:
python setup.py build_sphinx
You can access the build files in the following directory relative to the project root:
build/sphinx/html/
## Build
pip install wheel
python setup.py sdist bdist_wheel
## Publication
NASA PDS packages can publish automatically using the [Roundup Action](https://github.com/NASA-PDS/roundup-action), which leverages GitHub Actions to perform automated continuous integration and continuous delivery. A default workflow that includes the Roundup is provided in the `.github/workflows/unstable-cicd.yaml` file. (Unstable here means an interim release.)
### Manual Publication
Create the package:
python setup.py bdist_wheel
Publish it as a Github release.
Publish on PyPI (you need a PyPI account and configure `$HOME/.pypirc`):
pip install twine
twine upload dist/*
Or publish on the Test PyPI (you need a Test PyPI account and configure `$HOME/.pypirc`):
pip install twine
twine upload --repository testpypi dist/*
## CI/CD
The template repository comes with our two "standard" CI/CD workflows, `stable-cicd` and `unstable-cicd`. The unstable build runs on any push to `main` (± ignoring changes to specific files) and the stable build runs on push of a release branch of the form `release/<release version>`. Both of these make use of our GitHub actions build step, [Roundup](https://github.com/NASA-PDS/roundup-action). The `unstable-cicd` will generate (and constantly update) a SNAPSHOT release. If you haven't done a formal software release you will end up with a `v0.0.0-SNAPSHOT` release (see NASA-PDS/roundup-action#56 for specifics).
Raw data
{
"_id": null,
"home_page": "https://github.com/NASA-PDS/registry-client",
"name": "pds.registry-client",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "pds, planetary data, various, other, keywords",
"author": "PDS Engineering Node",
"author_email": "pds_operator@jpl.nasa.gov",
"download_url": "https://github.com/NASA-PDS/registry-client/releases/",
"platform": null,
"description": "\n# PDS Registry Client\n\nThis is a prototype implementation of a request-signing utility for use with serverless OpenSearch (AOSS). It is\n(currently) intended to provide a curl-like interface for querying PDS Registry's AOSS instance using a Cognito user\nidentity.\n\nAdditional functionality may be built out in the future.\n\n## Prerequisites\n\n- Personal user/pass credentials for a Cognito user authorized for Registry AOSS\n- Python >=3.9\n- Environment variables (contact developer for values)\n ```bash\n export REQUEST_SIGNER_AWS_ACCOUNT=''\n export REQUEST_SIGNER_AWS_REGION=''\n export REQUEST_SIGNER_CLIENT_ID=''\n export REQUEST_SIGNER_USER_POOL_ID=''\n export REQUEST_SIGNER_IDENTITY_POOL_ID=''\n export REQUEST_SIGNER_AOSS_ENDPOINT=''\n export REQUEST_SIGNER_COGNITO_USER=''\n export REQUEST_SIGNER_COGNITO_PASSWORD=''\n ```\n\n## Developer Quickstart\n\n1. Clone the repository\n ```\n git clone https://github.com/NASA-PDS/registry-client.git\n cd registry-client\n ```\n\n\n2. Create a virtual environment\n ```\n python -m venv venv\n source ./venv/bin/activate\n ```\n\n3. Install the tool to the virtual environment\n ```\n pip install --editable .[dev]\n ```\n\n4. Run the tool directly\n ```\n registry-client --help\n ```\n\n## Code of Conduct\n\nAll users and developers of the NASA-PDS software are expected to abide by our [Code of Conduct](https://github.com/NASA-PDS/.github/blob/main/CODE_OF_CONDUCT.md). Please read this to ensure you understand the expectations of our community.\n\n\n## Development\n\nTo develop this project, use your favorite text editor, or an integrated development environment with Python support, such as [PyCharm](https://www.jetbrains.com/pycharm/).\n\n\n### Contributing\n\nFor information on how to contribute to NASA-PDS codebases please take a look at our [Contributing guidelines](https://github.com/NASA-PDS/.github/blob/main/CONTRIBUTING.md).\n\n\n### Installation\n\nInstall in editable mode and with extra developer dependencies into your virtual environment of choice:\n\n pip install --editable '.[dev]'\n\nMake a baseline for any secrets (email addresses, passwords, API keys, etc.) in the repository:\n\n detect-secrets scan . \\\n --all-files \\\n --disable-plugin AbsolutePathDetectorExperimental \\\n --exclude-files '\\.secrets..*' \\\n --exclude-files '\\.git.*' \\\n --exclude-files '\\.mypy_cache' \\\n --exclude-files '\\.pytest_cache' \\\n --exclude-files '\\.tox' \\\n --exclude-files '\\.venv' \\\n --exclude-files 'venv' \\\n --exclude-files 'dist' \\\n --exclude-files 'build' \\\n --exclude-files '.*\\.egg-info' > .secrets.baseline\n\nReview the secrets to determine which should be allowed and which are false positives:\n\n detect-secrets audit .secrets.baseline\n\nPlease remove any secrets that should not be seen by the public. You can then add the baseline file to the commit:\n\n git add .secrets.baseline\n\nThen, configure the `pre-commit` hooks:\n\n pre-commit install\n pre-commit install -t pre-push\n pre-commit install -t prepare-commit-msg\n pre-commit install -t commit-msg\n\nThese hooks then will check for any future commits that might contain secrets. They also check code formatting, PEP8 compliance, type hints, etc.\n\n\ud83d\udc49 **Note:** A one time setup is required both to support `detect-secrets` and in your global Git configuration. See [the wiki entry on Secrets](https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Git-and-Github-Guide#detect-secrets) to learn how.\n\n\n### Packaging\n\nTo isolate and be able to re-produce the environment for this package, you should use a [Python Virtual Environment](https://docs.python.org/3/tutorial/venv.html). To do so, run:\n\n python -m venv venv\n\nThen exclusively use `venv/bin/python`, `venv/bin/pip`, etc.\n\nIf you have `tox` installed and would like it to create your environment and install dependencies for you run:\n\n tox --devenv <name you'd like for env> -e dev\n\nDependencies for development are specified as the `dev` `extras_require` in `setup.cfg`; they are installed into the virtual environment as follows:\n\n pip install --editable '.[dev]'\n\nAll the source code is in a sub-directory under `src`.\n\nYou should update the `setup.cfg` file with:\n\n- name of your module\n- license, default apache, update if needed\n- description\n- download url, when you release your package on github add the url here\n- keywords\n- classifiers\n- install_requires, add the dependencies of you package\n- extras_require, add the development Dependencies of your package\n- entry_points, when your package can be called in command line, this helps to deploy command lines entry points pointing to scripts in your package\n\nFor the packaging details, see https://packaging.python.org/tutorials/packaging-projects/ as a reference.\n\n\n### Configuration\n\nIt is convenient to use ConfigParser package to manage configuration. It allows a default configuration which can be overwritten by the user in a specific file in their environment. See https://pymotw.com/2/ConfigParser/\n\nFor example:\n\n candidates = ['my_pds_module.ini', 'my_pds_module.ini.default']\n found = parser.read(candidates)\n\n\n### Logs\n\nYou should not use `print()`vin the purpose of logging information on the execution of your code. Depending on where the code runs these information could be redirected to specific log files.\n\nTo make that work, start each Python file with:\n\n```python\n\"\"\"My module.\"\"\"\nimport logging\n\nlogger = logging.getLogger(__name__)\n```\n\nTo log a message:\n\n logger.info(\"my message\")\n\nIn your `main` routine, include:\n\n logging.basicConfig(level=logging.INFO)\n\nto get a basic logging system configured.\n\n\n### Tooling\n\nThe `dev` `extras_require` included in the template repo installs `black`, `flake8` (plus some plugins), and `mypy` along with default configuration for all of them. You can run all of these (and more!) with:\n\n tox -e lint\n\n\n### Code Style\n\nSo that your code is readable, you should comply with the [PEP8 style guide](https://www.python.org/dev/peps/pep-0008/). Our code style is automatically enforced in via [black](https://pypi.org/project/black/) and [flake8](https://flake8.pycqa.org/en/latest/). See the [Tooling section](#-tooling) for information on invoking the linting pipeline.\n\n\u2757Important note for template users\u2757\nThe included [pre-commit configuration file](.pre-commit-config.yaml) executes `flake8` (along with `mypy`) across the entire `src` folder and not only on changed files. If you're converting a pre-existing code base over to this template that may result in a lot of errors that you aren't ready to deal with.\n\nYou can instead execute `flake8` only over a diff of the current changes being made by modifying the `pre-commit` `entry` line:\n\n entry: git diff -u | flake8 --diff\n\nOr you can change the `pre-commit` config so `flake8` is only called on changed files which match a certain filtering criteria:\n\n - repo: local\n hooks:\n - id: flake8\n name: flake8\n entry: flake8\n files: ^src/|tests/\n language: system\n\n\n### Recommended Libraries\n\nPython offers a large variety of libraries. In PDS scope, for the most current usage we should use:\n\n| Library | Usage |\n|--------------|------------------------------------------------ |\n| configparser | manage and parse configuration files |\n| argparse | command line argument documentation and parsing |\n| requests | interact with web APIs |\n| lxml | read/write XML files |\n| json | read/write JSON files |\n| pyyaml | read/write YAML files |\n| pystache | generate files from templates |\n\nSome of these are built into Python 3; others are open source add-ons you can include in your `requirements.txt`.\n\n\n### Tests\n\nThis section describes testing for your package.\n\nA complete \"build\" including test execution, linting (`mypy`, `black`, `flake8`, etc.), and documentation build is executed via:\n\n tox\n\n\n#### Unit tests\n\nYour project should have built-in unit tests, functional, validation, acceptance, etc., tests.\n\nFor unit testing, check out the [unittest](https://docs.python.org/3/library/unittest.html) module, built into Python 3.\n\nTests objects should be in packages `test` modules or preferably in project 'tests' directory which mirrors the project package structure.\n\nOur unit tests are launched with command:\n\n pytest\n\nIf you want your tests to run automatically as you make changes start up `pytest` in watch mode with:\n\n ptw\n\n\n#### Integration/Behavioral Tests\n\nOne should use the `behave package` and push the test results to \"testrail\".\n\nSee an example in https://github.com/NASA-PDS/pds-doi-service#behavioral-testing-for-integration--testing\n\n\n### Documentation\n\nYour project should use [Sphinx](https://www.sphinx-doc.org/en/master/) to build its documentation. PDS' documentation template is already configured as part of the default build. You can build your projects docs with:\n\n python setup.py build_sphinx\n\nYou can access the build files in the following directory relative to the project root:\n\n build/sphinx/html/\n\n\n## Build\n\n pip install wheel\n python setup.py sdist bdist_wheel\n\n\n## Publication\n\nNASA PDS packages can publish automatically using the [Roundup Action](https://github.com/NASA-PDS/roundup-action), which leverages GitHub Actions to perform automated continuous integration and continuous delivery. A default workflow that includes the Roundup is provided in the `.github/workflows/unstable-cicd.yaml` file. (Unstable here means an interim release.)\n\n\n### Manual Publication\n\nCreate the package:\n\n python setup.py bdist_wheel\n\nPublish it as a Github release.\n\nPublish on PyPI (you need a PyPI account and configure `$HOME/.pypirc`):\n\n pip install twine\n twine upload dist/*\n\nOr publish on the Test PyPI (you need a Test PyPI account and configure `$HOME/.pypirc`):\n\n pip install twine\n twine upload --repository testpypi dist/*\n\n## CI/CD\n\nThe template repository comes with our two \"standard\" CI/CD workflows, `stable-cicd` and `unstable-cicd`. The unstable build runs on any push to `main` (\u00b1 ignoring changes to specific files) and the stable build runs on push of a release branch of the form `release/<release version>`. Both of these make use of our GitHub actions build step, [Roundup](https://github.com/NASA-PDS/roundup-action). The `unstable-cicd` will generate (and constantly update) a SNAPSHOT release. If you haven't done a formal software release you will end up with a `v0.0.0-SNAPSHOT` release (see NASA-PDS/roundup-action#56 for specifics).\n",
"bugtrack_url": null,
"license": "apache-2.0",
"summary": "A simple PDS Registry Client which authenticates users with PDS single sign-on and signs requests to the serverless OpenSearch (AOSS) hosting the Registry database.",
"version": "0.3.0",
"project_urls": {
"Download": "https://github.com/NASA-PDS/registry-client/releases/",
"Homepage": "https://github.com/NASA-PDS/registry-client"
},
"split_keywords": [
"pds",
" planetary data",
" various",
" other",
" keywords"
],
"urls": [
{
"comment_text": "\ud83e\udd20 Yee-haw! This here ar-tee-fact got done uploaded by the Roundup!",
"digests": {
"blake2b_256": "b1f2ada4191c0d8730e838647b3d049016f5856ef927fbb950a4673c94756f75",
"md5": "7a2efdf51925ba5b966b274503b087b8",
"sha256": "870272a1f10e7aeed6cdeae5efe1005c23b70a8604f0456346c1c26dc8f2cb87"
},
"downloads": -1,
"filename": "pds.registry_client-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7a2efdf51925ba5b966b274503b087b8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 17180,
"upload_time": "2024-10-15T21:55:43",
"upload_time_iso_8601": "2024-10-15T21:55:43.285580Z",
"url": "https://files.pythonhosted.org/packages/b1/f2/ada4191c0d8730e838647b3d049016f5856ef927fbb950a4673c94756f75/pds.registry_client-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-15 21:55:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "NASA-PDS",
"github_project": "registry-client",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pds.registry-client"
}
JSON
