# Cloudmesh Catalog
Cloudmesh catalog can be used to store information about a service, software
component, or project. The information included in it can be
categorized so that a comparision is possible. The catalog is
implemented as REST service so it can be integrated in other projects
and searched programatically.
The catalog depends on the cloudmesh command shell which allows eay
integration of new commands line environment. It projects a sample
Interface for the catalog from the commandline
We also can create static web pages from the catalog while using the export
feature and integrating the pages in for example hugo.
We are currently exploring hugo docsy as it provides
an easy way to generate hirarchical web pages, but also leverages
hugos tags and categories. Other export formats include markdown and
bibtex.
## Instalation for developers
1. If you do not have yet create an ssh key and upload it to the
github repository.
```ssh-keygen```
Upload the `~/.ssh/id_rsa.pub` key to github
2. Download cloudmesh with its source repositories
Make sure you ave python 3.10.2
On Mac or Linux do
```bash
$ python3.10 -m venv ~/ENV3
$ source ~/ENV3/bin/activate
```
On Windows
```bash
$ py --version # make sure its 3.10.2
$ py -m venv ~/ENV3
$ source ~/ENV3/bin/activate
```
After that the instalation is the same on all operating systems.
```bash
$ mkdir cm
$ cd cm
$ pip install cloudmesh-installer
$ cloudmesh-installer -ssh install catalog
$ cms help
```
This will download all source code for the cloudmesh shell
and compile from source.
3. Now you are all ready to do programming and enhancing
cloudmesh-catalog If you have any issues, contact
laszewski@gmail.com
## Manual page
A manual pasge shoudl be implemented in
`cloudmesh-catalog/catalog/command/catalog.py` This manual page can
be displayed with the following command:
```bash
$ cms help catalog help
```
To just see the usage type in
```bash
$ cms catalog
```
## Managing the Service
TODO: The integration of data into the service is not yet completed.
TODO: service management on Windows is not yet completed.
On Linux and macOS we can already experiment with an early prototype
that allows us starting, sopping, and getting the status of the service.
This service has nnot yet been integrated with a database.
## BUG
TODO: The adat is not yet integrated and we like to use
cloudmesh/yamldb for it.
## Adding catalog and registry data
TODO: To add catalog and registry data for new services, one must create new
.yaml files in the appropriate folders: 'data/catalog/my_example.yaml'
and 'data/registry/my_example.yaml'. Each file must follow yaml
formatting similar to the following example.
Example file: Amazon Comprehend (Catalog), amazon_comprehend.yaml
```
---
id: amazon_comprehend:
name: Amazon Comprehend
title: Amazon Comprehend
author: Amazon
slug: amazon-comprehend
public: true
description: |
Comprehend is Amazon's solution for cloud-based NLP.
It is available with an AWS account. To use,
it requires use of either the AWS Command Line
Interface or an AWS SDK for Python, Java, or .NET.
Notable features include functionality for giving
batches of documents to be processed as well as
submission of multiple jobs in a list. The DetectEntities
function also allows use of a custom-trained
model, but many other functions do not.
version: unknown
license: unknown
microservice: no
protocol: AWS API
owner: Amazon Web Services
modified: 9/29/2021
created: 11/29/2017
documentation: https://docs.aws.amazon.com/comprehend/index.html
source: unknown
specification: unknown
tags: ["nlp", "nlp service", "machine learning", "cloud service", "nlp api",
"deep learning", "natural language processing", "artificial intelligence"]
categories: ["NLP"]
additional_metadata: unknown
endpoint: unknown
sla: https://aws.amazon.com/machine-learning/language/sla/
authors: The AWS team can be contacted through support ticket at https://aws.amazon.com/contact-us/
data: |
User data is stored on Amazon servers under the associated AWS account and is protected under the AWS
shared responsibility model as detailed here https://aws.amazon.com/compliance/shared-responsibility-model/
```
## Using the Catalog and Registry classes
Written in catalog.py and registry.py are classes capable of reading and storing the data written in the .yaml files. Both use the same interface.
Here is an example of the Catalog class in action:
```
# initialize the catalog using data found in the given directory
catalog = Catalog('data/catalog/')
# query the catalog for Amazon Comprehend data, save result to amazon_catalog_data
amazon_catalog_data = cat.query({'name': 'Amazon Comprehend'})
# add a new data file to the catalog
catalog.add('new_example/azure_language.yaml')
# save entire catalog to a pickle file
catalog.to_pickle('catalog.pkl')
# load from pickle file
catalog.from_pickle('catalog.pkl')
# print catalog data
print(catalog.data)
```
## Using the yaml to markdown conversion script
The catalog command includes several prototype export formats
that takes all files recursively in a directory or an explicit file and converts it to the specified output
This includes
```bash
cms catalog export bibtex --souce=SOURCE
cms catalog export hugo --souce=SOURCE
cms catalog export md --souce=SOURCE
```
The commands will create next to the yal file entreies for bibtex,
hugo markdown, and markdown.
The templates are just suggestions and we may improve them based on our
findings.
## Checking entries
It is very important that any entry be checked for minimal yaml complience.
Hence we implemented a command
```bash
cms catalog check --souce=SOURCE
```
which will check all file sin the specified directory. THIs check will ignore line legth limits if the line contains an http or https refernce.
We also check the data format for YYYY-MM-DD.
We know that it may be problematic to distingush automatically between YYYY-MM-DD and YYYY-DD-MM.
Hence we encourage you to be careful when adding entries.
## Developer Video Tutorials
We are providing a number of developer video tutorials that help undesrtanding how we develop code and leverage the
cloudmesh-cmd5 shell features:
* [Cloudmesh Catalog. Who to improve the check feature](https://www.youtube.com/watch?v=OkuYxky2TFo)
* [Cloudmesh Catalog. Overview of the converter](https://www.youtube.com/watch?v=4bKvA3RhWaU)
* [Cloudmesh Catalog. How to use the integration with hugo](https://www.youtube.com/watch?v=LfC5JDddwLI)
* [Cloudmesh Catalog. Managing the server with start, stop, info, status](https://www.youtube.com/watch?v=qr8Yf5qAmV8)
* [Cloudmesh Catalog. Running the Server on a Mac on port 8001](https://www.youtube.com/watch?v=T2im0MeDDKE)
* [Github Tips and Project management](https://www.youtube.com/watch?v=Jce1kYOkd04)
* [Overview Cloudmesh NIST project](https://www.youtube.com/watch?v=BCIE44MDgwE)
Other videos are available at
* <https://www.youtube.com/c/GregorvonLaszewski/videos>
---
Raw data
{
"_id": null,
"home_page": "",
"name": "cloudmesh-catalog",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Gregor von Laszewski <laszewski@gmail.com>",
"keywords": "helper library,cloudmesh",
"author": "",
"author_email": "Gregor von Laszewski <laszewski@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/2e/c3/6a70a122cf9825e8b3e533b1072c5e24fcd336b3ceb625f96f65778a6128/cloudmesh-catalog-5.0.2.tar.gz",
"platform": null,
"description": "# Cloudmesh Catalog\n\nCloudmesh catalog can be used to store information about a service, software\ncomponent, or project. The information included in it can be\ncategorized so that a comparision is possible. The catalog is\nimplemented as REST service so it can be integrated in other projects\nand searched programatically.\n\nThe catalog depends on the cloudmesh command shell which allows eay\nintegration of new commands line environment. It projects a sample\nInterface for the catalog from the commandline\n\nWe also can create static web pages from the catalog while using the export \nfeature and integrating the pages in for example hugo.\n\nWe are currently exploring hugo docsy as it provides\nan easy way to generate hirarchical web pages, but also leverages \nhugos tags and categories. Other export formats include markdown and \nbibtex.\n\n## Instalation for developers\n\n1. If you do not have yet create an ssh key and upload it to the\n github repository.\n\n ```ssh-keygen```\n\n Upload the `~/.ssh/id_rsa.pub` key to github\n\n2. Download cloudmesh with its source repositories\n\n Make sure you ave python 3.10.2\n\n On Mac or Linux do\n\n ```bash\n $ python3.10 -m venv ~/ENV3\n $ source ~/ENV3/bin/activate\n ```\n\n On Windows \n\n ```bash\n $ py --version # make sure its 3.10.2\n $ py -m venv ~/ENV3\n $ source ~/ENV3/bin/activate\n ```\n\n After that the instalation is the same on all operating systems.\n\n ```bash\n $ mkdir cm\n $ cd cm\n \n $ pip install cloudmesh-installer\n $ cloudmesh-installer -ssh install catalog\n $ cms help\n ```\n \n This will download all source code for the cloudmesh shell\n and compile from source.\n\n3. Now you are all ready to do programming and enhancing\n cloudmesh-catalog If you have any issues, contact\n laszewski@gmail.com\n\n## Manual page\n\nA manual pasge shoudl be implemented in\n`cloudmesh-catalog/catalog/command/catalog.py` This manual page can\nbe displayed with the following command:\n\n```bash\n$ cms help catalog help\n```\n\nTo just see the usage type in \n\n```bash\n$ cms catalog\n```\n\n\n## Managing the Service\n\nTODO: The integration of data into the service is not yet completed.\n\nTODO: service management on Windows is not yet completed.\n\nOn Linux and macOS we can already experiment with an early prototype \nthat allows us starting, sopping, and getting the status of the service. \nThis service has nnot yet been integrated with a database.\n\n\n## BUG\n\nTODO: The adat is not yet integrated and we like to use \ncloudmesh/yamldb for it.\n\n## Adding catalog and registry data\n\nTODO: To add catalog and registry data for new services, one must create new\n.yaml files in the appropriate folders: 'data/catalog/my_example.yaml'\nand 'data/registry/my_example.yaml'. Each file must follow yaml\nformatting similar to the following example.\n\nExample file: Amazon Comprehend (Catalog), amazon_comprehend.yaml\n\n```\n---\nid: amazon_comprehend:\nname: Amazon Comprehend\ntitle: Amazon Comprehend\nauthor: Amazon\nslug: amazon-comprehend\npublic: true\ndescription: |\n Comprehend is Amazon's solution for cloud-based NLP.\n It is available with an AWS account. To use,\n it requires use of either the AWS Command Line\n Interface or an AWS SDK for Python, Java, or .NET.\n Notable features include functionality for giving\n batches of documents to be processed as well as\n submission of multiple jobs in a list. The DetectEntities\n function also allows use of a custom-trained\n model, but many other functions do not.\nversion: unknown\nlicense: unknown\nmicroservice: no\nprotocol: AWS API\nowner: Amazon Web Services\nmodified: 9/29/2021\ncreated: 11/29/2017\ndocumentation: https://docs.aws.amazon.com/comprehend/index.html\nsource: unknown\nspecification: unknown\ntags: [\"nlp\", \"nlp service\", \"machine learning\", \"cloud service\", \"nlp api\",\n \"deep learning\", \"natural language processing\", \"artificial intelligence\"]\ncategories: [\"NLP\"]\nadditional_metadata: unknown\nendpoint: unknown\nsla: https://aws.amazon.com/machine-learning/language/sla/\nauthors: The AWS team can be contacted through support ticket at https://aws.amazon.com/contact-us/\ndata: |\n User data is stored on Amazon servers under the associated AWS account and is protected under the AWS\n shared responsibility model as detailed here https://aws.amazon.com/compliance/shared-responsibility-model/\n```\n\n## Using the Catalog and Registry classes\n\nWritten in catalog.py and registry.py are classes capable of reading and storing the data written in the .yaml files. Both use the same interface.\nHere is an example of the Catalog class in action:\n\n```\n# initialize the catalog using data found in the given directory\ncatalog = Catalog('data/catalog/')\n# query the catalog for Amazon Comprehend data, save result to amazon_catalog_data\namazon_catalog_data = cat.query({'name': 'Amazon Comprehend'})\n# add a new data file to the catalog\ncatalog.add('new_example/azure_language.yaml')\n# save entire catalog to a pickle file\ncatalog.to_pickle('catalog.pkl')\n# load from pickle file\ncatalog.from_pickle('catalog.pkl')\n# print catalog data\nprint(catalog.data)\n```\n\n## Using the yaml to markdown conversion script\n\nThe catalog command includes several prototype export formats \nthat takes all files recursively in a directory or an explicit file and converts it to the specified output \n\nThis includes \n\n```bash\ncms catalog export bibtex --souce=SOURCE\ncms catalog export hugo --souce=SOURCE\ncms catalog export md --souce=SOURCE\n```\n\nThe commands will create next to the yal file entreies for bibtex, \nhugo markdown, and markdown.\n\nThe templates are just suggestions and we may improve them based on our\nfindings.\n\n## Checking entries\n\nIt is very important that any entry be checked for minimal yaml complience.\nHence we implemented a command \n\n```bash\ncms catalog check --souce=SOURCE\n```\n\nwhich will check all file sin the specified directory. THIs check will ignore line legth limits if the line contains an http or https refernce. \nWe also check the data format for YYYY-MM-DD.\n\nWe know that it may be problematic to distingush automatically between YYYY-MM-DD and YYYY-DD-MM.\nHence we encourage you to be careful when adding entries.\n\n\n## Developer Video Tutorials\n\nWe are providing a number of developer video tutorials that help undesrtanding how we develop code and leverage the \ncloudmesh-cmd5 shell features:\n\n* [Cloudmesh Catalog. Who to improve the check feature](https://www.youtube.com/watch?v=OkuYxky2TFo)\n* [Cloudmesh Catalog. Overview of the converter](https://www.youtube.com/watch?v=4bKvA3RhWaU)\n* [Cloudmesh Catalog. How to use the integration with hugo](https://www.youtube.com/watch?v=LfC5JDddwLI)\n* [Cloudmesh Catalog. Managing the server with start, stop, info, status](https://www.youtube.com/watch?v=qr8Yf5qAmV8)\n* [Cloudmesh Catalog. Running the Server on a Mac on port 8001](https://www.youtube.com/watch?v=T2im0MeDDKE)\n* [Github Tips and Project management](https://www.youtube.com/watch?v=Jce1kYOkd04)\n* [Overview Cloudmesh NIST project](https://www.youtube.com/watch?v=BCIE44MDgwE)\n\nOther videos are available at\n\n* <https://www.youtube.com/c/GregorvonLaszewski/videos>\n\n\n---\n",
"bugtrack_url": null,
"license": "Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ Copyright 2017-2021 Gregor von Laszewski, Indiana University Copyright 2021,2022 Gregor von Laszewski, University of Virginia Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ",
"summary": "An example cloudmesh command",
"version": "5.0.2",
"project_urls": {
"Changelog": "https://github.com/cloudmesh/cloudmesh-catalog/blob/main/CHANGELOG.md",
"Documentation": "https://github.com/cloudmesh/cloudmesh-catalog/blob/main/README.md",
"Homepage": "https://github.com/cloudmesh/cloudmesh-catalog",
"Issues": "https://github.com/cloudmesh/cloudmesh-catalog/issues",
"Repository": "https://github.com/cloudmesh/cloudmesh-catalog.git"
},
"split_keywords": [
"helper library",
"cloudmesh"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b7a0188e99b62ea80c83a9207f0dbf2c942fceb1bd6b5478104fd08d5e1da10f",
"md5": "543bb685d4043b4fe2dbf81630717557",
"sha256": "12ffe701a66ad056970f151a8115d099260150179445c35b18e429c4375dc2df"
},
"downloads": -1,
"filename": "cloudmesh_catalog-5.0.2-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "543bb685d4043b4fe2dbf81630717557",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 22712,
"upload_time": "2023-12-23T05:37:48",
"upload_time_iso_8601": "2023-12-23T05:37:48.498825Z",
"url": "https://files.pythonhosted.org/packages/b7/a0/188e99b62ea80c83a9207f0dbf2c942fceb1bd6b5478104fd08d5e1da10f/cloudmesh_catalog-5.0.2-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2ec36a70a122cf9825e8b3e533b1072c5e24fcd336b3ceb625f96f65778a6128",
"md5": "2ea548fd06d3ee9c5fe289034d1ad553",
"sha256": "1ea8891a5e00079959930d7833b429a0fe3b25720708840e1e299b2f5796ed04"
},
"downloads": -1,
"filename": "cloudmesh-catalog-5.0.2.tar.gz",
"has_sig": false,
"md5_digest": "2ea548fd06d3ee9c5fe289034d1ad553",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 24568,
"upload_time": "2023-12-23T05:37:50",
"upload_time_iso_8601": "2023-12-23T05:37:50.712892Z",
"url": "https://files.pythonhosted.org/packages/2e/c3/6a70a122cf9825e8b3e533b1072c5e24fcd336b3ceb625f96f65778a6128/cloudmesh-catalog-5.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-23 05:37:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cloudmesh",
"github_project": "cloudmesh-catalog",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "cloudmesh-catalog"
}
