======
README
======
Flexible Dataset Integrator (FDI) helps data producers and processors to build connections into isolated heterogeneous datasets. to assemble, organize, and integrate data into self-describing, modular, hierarchical, persistent, referenceable ``Products``, whose component datasets keep their own characteristicss and are easily accessible.
FDI provides scripting-friendly APIs and
tools to define custom Products and generating Python class files. An integrated Product takes care of inter-platform compatibility, string representation, serialisation to simple exchange format, persistence to disk or server, and carrying references of other Products, enabling context-building and lazy-loading.
FDI's base data model is defined in sub-package ``dataset``. Persistent data
access using storage pools, referencing, and Universal Resource Names, and referencie-carrying product Context, are defined in sub-package ``pal``. A reference RESTful API server designed to communicate with a data processing server/docker using the data model, and a reference HTTP pool server are in sub-package ``pns``.
Install/Uninstall
=================
.. image:: pipeline.svg
:target: http://mercury.bao.ac.cn:9006/mh/fdi/badges/master/index.html
<a href="http://mercury.bao.ac.cn:9006/mh/fdi/-/commits/master"><img alt="pipeline status" src="http://mercury.bao.ac.cn:9006/mh/fdi/badges/master/pipeline.svg" /></a>
.. tip::
It is a good idea to add ``--user`` at the end or to use a virtualenv to avoid disturbing your system Python setup.
For Users
---------
If you need to use fdi, install from PyPI:
.. code-block:: shell
python3 -m pip install fdi
or from git repo:
.. code-block:: shell
python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/master/fdi-master.tar.gz
If you want to install the ``develop`` dependencies:
.. code-block:: shell
python3 -m pip install fdi[DEV]
or
.. code-block:: shell
python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/develop/fdi-develop.tar.gz#egg=fdi[DEV]
If you do not need to run tests, remove ``[DEV]`` to save time and disk space.
To uninstall:
.. code-block:: shell
python3 -m pip uninstall fdi
For Developers (or Those who are Not Sure which to Choose)
-----------------------------------------------------------
To install
''''''''''
.. code-block:: shell
FDIINSTDIR=/tmp # change this to your installation dir
cd $FDIINSTDIR
git clone http://mercury.bao.ac.cn:9006/mh/fdi.git
cd fdi
git checkout develop
make install EXT="[DEV]"
If you want to install the ``master`` branch, remove the ``git checkout develop`` line above.
To test your installation
'''''''''''''''''''''''''
.. code-block:: shell
make test
.. tip::
To pass command-line arguments to ``pytest`` do, for example,
.. code-block:: shell
make test T='-k Bas'
to test ``BaseProduct``.
To Generate Python Product Source Code
--------------------------------------
Re-generate ``baseproduct.py`` and ``product.py`` from YAML schema files in
``fdi/dataset/resources``:
.. code-block:: shell
make py
Learn/Try the HTTP Pool Server APIs
-----------------------------------
The APIs are documented in `fdi/httppool/schema/pools.yml` with OpenAPI 3. Run this to see and try out with Swagger API Docs when the server is running:
.. code-block:: shell
http://127.0.0.1:5000/apidocs
Modify/Generate Documents
-------------------------
If you plan to compile documents in the ``docs`` directory, generate diagrams, API files, or HTML pages, run (in that order, respectively):
First run this once to install necessary packages:
.. code-block:: shell
python3 -m pip install -U -e .[PUB]
Then when you need to make new class diagrams, API docs, or HTML pages:
.. code-block:: shell
make docs_plots
make docs_api
make docs_html
The generated HTML page is at ``docs/html/index.html``.
.. note:: https://readthedocs.io makes web pages from sources in ``docs/sphinx`` in the repository. Locally generated HTML pages are not on RTD or in the repository. The API files and plots, however, are in ``api`` and ``_static`` sub-directpries, respectively.
Run Servers and Quick Tests
---------------------------
If you plan to run the ``pns`` and/or the ``http pool server`` locally,
first install the dependencies:
.. code-block:: shell
python3 -m pip install -e .[SERV]
make installpns
To test your ``pns`` servers installation, in one window, run:
.. code-block:: shell
make runserver
in another window run:
.. code-block:: shell
make testpns
To test your ``httppool`` servers installation, make sure to stop other server such as ``pnsserver`` above, then in one window, run:
.. code-block:: shell
make runpoolserver
in another window run:
.. code-block:: shell
make testhttp
Docker Containers
-----------------
fdi
'''
Get the ``fdi`` docker with running FDI environment:
.. code-block:: shell
docker pull mhastro/fdi
or build the image locally:
.. code-block:: shell
make build_docker
Launch and login
.. code-block:: shell
make launch_docker
make it
httppool
''''''''
Also available is a ``HttpPool`` server made from Ubuntu and apache:
.. code-block:: shell
docker pull mhastro/httppool
To build the image locally:
.. code-block:: shell
make build_server
Launch and connect:
.. code-block:: shell
make launch_server
curl -i http://127.0.0.1:9884/v0.8/
Run the above in the package root directory of fdi. A file named ``.secret`` is needed by the build and launch commands. This is an example::
PNS_PORT=9884
PNS_USERNAME=...
PNS__PASSWORD=...
PNS_MQ_HOST=123.45.67.89
PNS_MQ_PORT=9876
PNS_MQ_USER=...
PNS_MQ_PASS=...
For More
--------
For more examples see ``tests/test_*.py``.
Read more on package introduction, description, quick start, and API
documents on `readthedocs.io <https://fdi.readthedocs.io/en/latest/>`__.
Raw data
{
"_id": null,
"home_page": "http://mercury.bao.ac.cn:9006/mh/fdi",
"name": "fdi",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "dataset metadata processing product context serialization server URN RESTful API HCSS",
"author": "Maohai Huang",
"author_email": "mhuang@earth.bao.ac.cn",
"download_url": "https://files.pythonhosted.org/packages/25/95/4e7a9410c0129273a03bd8f31f7d02d0b3eb3cfa413b254bcf2222f625e4/fdi-1.34.2.tar.gz",
"platform": null,
"description": "======\nREADME\n======\n\nFlexible Dataset Integrator (FDI) helps data producers and processors to build connections into isolated heterogeneous datasets. to assemble, organize, and integrate data into self-describing, modular, hierarchical, persistent, referenceable ``Products``, whose component datasets keep their own characteristicss and are easily accessible.\n\nFDI provides scripting-friendly APIs and \ntools to define custom Products and generating Python class files. An integrated Product takes care of inter-platform compatibility, string representation, serialisation to simple exchange format, persistence to disk or server, and carrying references of other Products, enabling context-building and lazy-loading.\n\nFDI's base data model is defined in sub-package ``dataset``. Persistent data\naccess using storage pools, referencing, and Universal Resource Names, and referencie-carrying product Context, are defined in sub-package ``pal``. A reference RESTful API server designed to communicate with a data processing server/docker using the data model, and a reference HTTP pool server are in sub-package ``pns``.\n\nInstall/Uninstall\n=================\n\n.. image:: pipeline.svg\n :target: http://mercury.bao.ac.cn:9006/mh/fdi/badges/master/index.html\n\n<a href=\"http://mercury.bao.ac.cn:9006/mh/fdi/-/commits/master\"><img alt=\"pipeline status\" src=\"http://mercury.bao.ac.cn:9006/mh/fdi/badges/master/pipeline.svg\" /></a>\n\n.. tip::\n\n It is a good idea to add ``--user`` at the end or to use a virtualenv to avoid disturbing your system Python setup.\n\nFor Users\n---------\n\nIf you need to use fdi, install from PyPI:\n\n.. code-block:: shell\n\n python3 -m pip install fdi\n\nor from git repo:\n\n.. code-block:: shell\n\n python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/master/fdi-master.tar.gz\n\nIf you want to install the ``develop`` dependencies:\n\n.. code-block:: shell\n\n python3 -m pip install fdi[DEV]\n \nor\n\n.. code-block:: shell\n\n python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/develop/fdi-develop.tar.gz#egg=fdi[DEV]\n\n\nIf you do not need to run tests, remove ``[DEV]`` to save time and disk space.\n\nTo uninstall:\n\n.. code-block:: shell\n\n python3 -m pip uninstall fdi\n\n\nFor Developers (or Those who are Not Sure which to Choose)\n-----------------------------------------------------------\n\nTo install\n''''''''''\n\n.. code-block:: shell\n\n FDIINSTDIR=/tmp # change this to your installation dir\n cd $FDIINSTDIR\n git clone http://mercury.bao.ac.cn:9006/mh/fdi.git\n cd fdi\n\t git checkout develop\n\t make install EXT=\"[DEV]\"\n\t \nIf you want to install the ``master`` branch, remove the ``git checkout develop`` line above.\n\t \nTo test your installation\n'''''''''''''''''''''''''\n\n.. code-block:: shell\n\n make test\n\n.. tip::\n\n To pass command-line arguments to ``pytest`` do, for example,\n \n .. code-block:: shell\n\t\t \n\t\tmake test T='-k Bas'\n\n to test ``BaseProduct``.\n\n\nTo Generate Python Product Source Code\n--------------------------------------\n\nRe-generate ``baseproduct.py`` and ``product.py`` from YAML schema files in\n``fdi/dataset/resources``:\n\n.. code-block:: shell\n\n make py\n\nLearn/Try the HTTP Pool Server APIs\n-----------------------------------\n\nThe APIs are documented in `fdi/httppool/schema/pools.yml` with OpenAPI 3. Run this to see and try out with Swagger API Docs when the server is running:\n\n.. code-block:: shell\n\n\t\thttp://127.0.0.1:5000/apidocs\n\nModify/Generate Documents\n-------------------------\n\nIf you plan to compile documents in the ``docs`` directory, generate diagrams, API files, or HTML pages, run (in that order, respectively):\n\nFirst run this once to install necessary packages:\n\n.. code-block:: shell\n\n python3 -m pip install -U -e .[PUB]\n\nThen when you need to make new class diagrams, API docs, or HTML pages:\n\n.. code-block:: shell\n\n make docs_plots\n make docs_api\n make docs_html\n\nThe generated HTML page is at ``docs/html/index.html``.\n\n.. note:: https://readthedocs.io makes web pages from sources in ``docs/sphinx`` in the repository. Locally generated HTML pages are not on RTD or in the repository. The API files and plots, however, are in ``api`` and ``_static`` sub-directpries, respectively.\n\t \nRun Servers and Quick Tests\n---------------------------\n\nIf you plan to run the ``pns`` and/or the ``http pool server`` locally,\nfirst install the dependencies:\n\n.. code-block:: shell\n\n python3 -m pip install -e .[SERV]\n\t make installpns\n\nTo test your ``pns`` servers installation, in one window, run:\n\n.. code-block:: shell\n\n make runserver\n\nin another window run:\n\n.. code-block:: shell\n\n make testpns\n\nTo test your ``httppool`` servers installation, make sure to stop other server such as ``pnsserver`` above, then in one window, run:\n\n.. code-block:: shell\n\n make runpoolserver\n\nin another window run:\n\n.. code-block:: shell\n\n make testhttp\n\nDocker Containers\n-----------------\n\nfdi\n'''\n\nGet the ``fdi`` docker with running FDI environment:\n\n.. code-block:: shell\n\n docker pull mhastro/fdi\n \nor build the image locally:\n\n.. code-block:: shell\n\n make build_docker\n\nLaunch and login\n\n.. code-block:: shell\n\n make launch_docker\n make it\n\nhttppool\n''''''''\n \nAlso available is a ``HttpPool`` server made from Ubuntu and apache:\n\n.. code-block:: shell\n\n docker pull mhastro/httppool\n \nTo build the image locally:\n\n.. code-block:: shell\n\n make build_server \n\t\t\nLaunch and connect:\n\n.. code-block:: shell\n\n make launch_server\n curl -i http://127.0.0.1:9884/v0.8/\n\nRun the above in the package root directory of fdi. A file named ``.secret`` is needed by the build and launch commands. This is an example::\n\n PNS_PORT=9884\n PNS_USERNAME=...\n PNS__PASSWORD=...\n PNS_MQ_HOST=123.45.67.89\n PNS_MQ_PORT=9876\n PNS_MQ_USER=...\n PNS_MQ_PASS=...\n\nFor More\n--------\n\nFor more examples see ``tests/test_*.py``.\n\nRead more on package introduction, description, quick start, and API\ndocuments on `readthedocs.io <https://fdi.readthedocs.io/en/latest/>`__.\n\n",
"bugtrack_url": null,
"license": "LGPL v3",
"summary": "Flexible Data Integrator",
"version": "1.34.2",
"project_urls": {
"Homepage": "http://mercury.bao.ac.cn:9006/mh/fdi"
},
"split_keywords": [
"dataset",
"metadata",
"processing",
"product",
"context",
"serialization",
"server",
"urn",
"restful",
"api",
"hcss"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "522bc9f926b3f3d84b7b2d9bd40d875f34c9c7bcbeabd568074e810520b7c550",
"md5": "60f78a32c8e91f2853d22c630245ba3d",
"sha256": "5f147dae0022ab5fbe18d03bc7deb51630e3b90d5c4d9050cb808538c43ee5f4"
},
"downloads": -1,
"filename": "fdi-1.34.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "60f78a32c8e91f2853d22c630245ba3d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 355646,
"upload_time": "2023-07-06T05:23:59",
"upload_time_iso_8601": "2023-07-06T05:23:59.276097Z",
"url": "https://files.pythonhosted.org/packages/52/2b/c9f926b3f3d84b7b2d9bd40d875f34c9c7bcbeabd568074e810520b7c550/fdi-1.34.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "25954e7a9410c0129273a03bd8f31f7d02d0b3eb3cfa413b254bcf2222f625e4",
"md5": "eec5651a5935d04a93a4978acee27291",
"sha256": "a5631e5201ec71cdf06db692d050f394d9e25c6a82980198639962eb74d27c26"
},
"downloads": -1,
"filename": "fdi-1.34.2.tar.gz",
"has_sig": false,
"md5_digest": "eec5651a5935d04a93a4978acee27291",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 3763210,
"upload_time": "2023-07-06T05:24:07",
"upload_time_iso_8601": "2023-07-06T05:24:07.617669Z",
"url": "https://files.pythonhosted.org/packages/25/95/4e7a9410c0129273a03bd8f31f7d02d0b3eb3cfa413b254bcf2222f625e4/fdi-1.34.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-06 05:24:07",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "fdi"
}
JSON