# Common Utils
[](https://github.com/gao-hongnan/common-utils/actions/workflows/continuous_integration.yaml)
## Continuous Integration
### Virtual Environment
First, make a virtual environment with `make_venv.sh`:
```bash
curl -s -o make_venv.sh \
https://raw.githubusercontent.com/gao-hongnan/common-utils/main/scripts/devops/make_venv.sh && \
bash make_venv.sh venv --pyproject --dev && \
source venv/bin/activate && \
rm make_venv.sh
```
### Continue on error vs If Always
See
[here](https://stackoverflow.com/questions/58858429/how-to-run-a-github-actions-step-even-if-the-previous-step-fails-while-still-f/58859404#58859404).
### Run Bandit Security Check
```bash
bash ./scripts/devops/ci/ci_security_bandit.sh \
--severity-level=low \
--format=json \
--output=bandit_results.json \
common_utils
```
### Run Linter Check
```bash
bash ./scripts/devops/ci/ci_linter_pylint.sh \
--rcfile=pyproject.toml \
--fail-under=10 \
--score=yes \
--output-format=json:pylint_results.json,colorized \
common_utils
```
### Run Formatter Black Check
```bash
bash ./scripts/devops/ci/ci_formatter_black.sh \
--check \
--diff \
--color \
--verbose \
common_utils
```
### Run Formatter Isort Check
```bash
bash ./scripts/devops/ci/ci_formatter_isort.sh \
--check \
--diff \
--color \
--verbose \
common_utils
```
## Run MyPy Type Check
```bash
bash ./scripts/devops/ci/ci_typing_mypy.sh \
--config-file=pyproject.toml \
common_utils \
| tee mypy_results.log
```
### Run Unit Test
## Run Integration Test
### Run System Test
### Run Acceptance Test
See [madewithml](https://madewithml.com/courses/mlops/testing/).
### Run Data Test (Great Expectations)
### Run Markdown Lint
```bash
npm install -g markdownlint-cli && \
touch .markdownlint.json && \
```
```bash
npm install --save-dev --save-exact prettier
```
```bash
# prettier
function pr() {
if [ -z "$1" ]; then
echo "Error: TARGET_DIR is mandatory."
return 1
fi
TARGET_DIR="$1"
prettier "$TARGET_DIR" --write \
--prose-wrap always \
--print-width 80 \
--tab-width 4 \
--use-tabs true
}
```
```bash
pr <TARGET_MARKDOWN_FILE>
markdownlint --fix <TARGET_MARKDOWN_FILE>
```
Raw data
{
"_id": null,
"home_page": "",
"name": "gaohn-common-utils",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "Gao Hongnan <hongnangao@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/a7/dd/9ee5730b25493d7cac5a948ae6f7033925a73ce77c40c081e2e81c982d8a/gaohn-common-utils-0.0.112.tar.gz",
"platform": null,
"description": "# Common Utils\n\n[](https://github.com/gao-hongnan/common-utils/actions/workflows/continuous_integration.yaml)\n\n## Continuous Integration\n\n### Virtual Environment\n\nFirst, make a virtual environment with `make_venv.sh`:\n\n```bash\ncurl -s -o make_venv.sh \\\n https://raw.githubusercontent.com/gao-hongnan/common-utils/main/scripts/devops/make_venv.sh && \\\nbash make_venv.sh venv --pyproject --dev && \\\nsource venv/bin/activate && \\\nrm make_venv.sh\n```\n\n### Continue on error vs If Always\n\nSee\n[here](https://stackoverflow.com/questions/58858429/how-to-run-a-github-actions-step-even-if-the-previous-step-fails-while-still-f/58859404#58859404).\n\n### Run Bandit Security Check\n\n```bash\nbash ./scripts/devops/ci/ci_security_bandit.sh \\\n --severity-level=low \\\n --format=json \\\n --output=bandit_results.json \\\n common_utils\n```\n\n### Run Linter Check\n\n```bash\nbash ./scripts/devops/ci/ci_linter_pylint.sh \\\n --rcfile=pyproject.toml \\\n --fail-under=10 \\\n --score=yes \\\n --output-format=json:pylint_results.json,colorized \\\n common_utils\n```\n\n### Run Formatter Black Check\n\n```bash\nbash ./scripts/devops/ci/ci_formatter_black.sh \\\n --check \\\n --diff \\\n --color \\\n --verbose \\\n common_utils\n```\n\n### Run Formatter Isort Check\n\n```bash\nbash ./scripts/devops/ci/ci_formatter_isort.sh \\\n --check \\\n --diff \\\n --color \\\n --verbose \\\n common_utils\n```\n\n## Run MyPy Type Check\n\n```bash\nbash ./scripts/devops/ci/ci_typing_mypy.sh \\\n --config-file=pyproject.toml \\\n common_utils \\\n | tee mypy_results.log\n```\n\n### Run Unit Test\n\n## Run Integration Test\n\n### Run System Test\n\n### Run Acceptance Test\n\nSee [madewithml](https://madewithml.com/courses/mlops/testing/).\n\n### Run Data Test (Great Expectations)\n\n### Run Markdown Lint\n\n```bash\nnpm install -g markdownlint-cli && \\\ntouch .markdownlint.json && \\\n```\n\n```bash\nnpm install --save-dev --save-exact prettier\n```\n\n```bash\n# prettier\nfunction pr() {\n if [ -z \"$1\" ]; then\n echo \"Error: TARGET_DIR is mandatory.\"\n return 1\n fi\n\n TARGET_DIR=\"$1\"\n prettier \"$TARGET_DIR\" --write \\\n --prose-wrap always \\\n --print-width 80 \\\n --tab-width 4 \\\n --use-tabs true\n}\n```\n\n```bash\npr <TARGET_MARKDOWN_FILE>\nmarkdownlint --fix <TARGET_MARKDOWN_FILE>\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "A small utility package",
"version": "0.0.112",
"project_urls": {
"Bug Tracker": "https://github.com/gao-hongnan/common-utils/issues",
"Homepage": "https://github.com/gao-hongnan/common-utils"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "82d7dd98d2c364fb795fd1a611754ba70e24ebbf72fada449704524255959e55",
"md5": "42f9fa9c6d8727a8b48f23a9d5d8fecc",
"sha256": "82e36614c4adb902ad5bc95092fd3076ace4ecd95acd6b5d0ccd9d8049f85722"
},
"downloads": -1,
"filename": "gaohn_common_utils-0.0.112-py3-none-any.whl",
"has_sig": false,
"md5_digest": "42f9fa9c6d8727a8b48f23a9d5d8fecc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 39655,
"upload_time": "2023-08-18T11:00:45",
"upload_time_iso_8601": "2023-08-18T11:00:45.397262Z",
"url": "https://files.pythonhosted.org/packages/82/d7/dd98d2c364fb795fd1a611754ba70e24ebbf72fada449704524255959e55/gaohn_common_utils-0.0.112-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a7dd9ee5730b25493d7cac5a948ae6f7033925a73ce77c40c081e2e81c982d8a",
"md5": "99e6f65a075068f27151ac00676a90a4",
"sha256": "6b90992bd1a93e66b6d83a8478094433e74e341107ab46e7e0639394779092a5"
},
"downloads": -1,
"filename": "gaohn-common-utils-0.0.112.tar.gz",
"has_sig": false,
"md5_digest": "99e6f65a075068f27151ac00676a90a4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 32170,
"upload_time": "2023-08-18T11:00:47",
"upload_time_iso_8601": "2023-08-18T11:00:47.097387Z",
"url": "https://files.pythonhosted.org/packages/a7/dd/9ee5730b25493d7cac5a948ae6f7033925a73ce77c40c081e2e81c982d8a/gaohn-common-utils-0.0.112.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-18 11:00:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "gao-hongnan",
"github_project": "common-utils",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "awscli",
"specs": []
},
{
"name": "db-dtypes",
"specs": [
[
"<",
"2.0"
],
[
">=",
"1.1"
]
]
},
{
"name": "google-cloud-bigquery",
"specs": [
[
">=",
"3.10"
],
[
"<",
"4.0"
]
]
},
{
"name": "google-cloud-storage",
"specs": [
[
">=",
"2.8"
],
[
"<",
"3.0"
]
]
},
{
"name": "ipython",
"specs": []
},
{
"name": "numpy",
"specs": [
[
"<",
"2.0"
],
[
">=",
"1.24"
]
]
},
{
"name": "pandas",
"specs": [
[
"<",
"2.0"
],
[
">=",
"1.5"
]
]
},
{
"name": "prettytable",
"specs": [
[
"<",
"4.0"
],
[
">=",
"3.7"
]
]
},
{
"name": "pydantic",
"specs": [
[
">=",
"2.0"
],
[
"<",
"3.0"
]
]
},
{
"name": "pytz",
"specs": [
[
">=",
"2023.3"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"<",
"2.0"
],
[
">=",
"1.0"
]
]
},
{
"name": "requests",
"specs": [
[
">=",
"2.28"
],
[
"<",
"3.0"
]
]
},
{
"name": "rich",
"specs": [
[
"<",
"14.0"
],
[
">=",
"13.3"
]
]
},
{
"name": "tabulate",
"specs": [
[
"<",
"1.0"
],
[
">=",
"0.9"
]
]
},
{
"name": "torch",
"specs": [
[
">=",
"2.0"
],
[
"<",
"3.0"
]
]
},
{
"name": "torchaudio",
"specs": [
[
">=",
"2.0"
],
[
"<",
"3.0"
]
]
},
{
"name": "torchvision",
"specs": [
[
"<",
"0.16"
],
[
">=",
"0.15"
]
]
}
],
"lcname": "gaohn-common-utils"
}
JSON
