=============
DCIC Snovault
=============
|Build status|_
.. |Build status| image:: https://travis-ci.org/4dn-dcic/snovault.svg?branch=master
.. _Build status: https://travis-ci.org/4dn-dcic/snovault
.. Important::
DCIC Snovault is a FORK of `snovault <https://pypi.org/project/snovault/>`_
created at the `ENCODE DCC project at Stanford <https://github.com/ENCODE-DCC>`_.
Our fork supports other projects of the
`4D Nucleome Data Coordination and Integration Center (4DN-DCIC)
<https://github.com/4dn-dcic>`_.
Although this software is available as open source software,
its primary function is to support our layered projects,
and we are not at this time able to offer any active support for other uses.
In particular, this fork does not purport to supersede
the original `snovault <https://pypi.org/project/snovault/>`_.
we just have a different use case that we are actively exploring.
Overview
========
DCIC Snovault is a JSON-LD Database Framework that serves as the backend for the 4DN Data portal and CGAP. Check out our full documentation `here
<https://snovault.readthedocs.io/en/latest/>`_.
.. note::
This repository contains a core piece of functionality shared amongst several projects
in the 4DN-DCIC. It is meant to be used internally by the DCIC team
in support of `Fourfront <https://data.4dnucleome.org>`_\ ,
the 4DN data portal, and at this point in time it is not expected to be useful
in a standalone/plug-and-play way to others.
Installation in 4DN components
==============================
DCIC Snovault is pip installable as the ``dcicsnovault`` package with::
$ pip install dcicsnovault``
However, at the present time, the functionality it provides might only be useful in conjunction
with other 4DN-DCIC components.
NOTE: If you'd like to enable Elasticsearch mapping with type=nested, set the environment variable "MAPPINGS_USE_NESTED"
or set the registry setting "mappings.use_nested".
Installation for Development
============================
Currently these are for Mac OSX using homebrew. If using linux, install dependencies with a different package manager.
Step 0: Install Xcode
---------------------
Install Xcode (from App Store) and homebrew: http://brew.sh
Step 1: Verify Homebrew Itself
------------------------------
Verify that homebrew is working properly::
$ brew doctor
Step 2: Install Homebrewed Dependencies
---------------------------------------
Install or update dependencies::
$ brew install libevent libmagic libxml2 libxslt openssl postgresql graphviz python3
$ brew install freetype libjpeg libtiff littlecms webp # Required by Pillow
$ brew cask install adoptopenjdk8
$ brew install elasticsearch@5.6
NOTES:
* If installation of adtopopenjdk8 fails due to an ambiguity, it should work to do this instead::
$ brew cask install homebrew/cask-versions/adoptopenjdk8
* If you try to invoke elasticsearch and it is not found,
you may need to link the brew-installed elasticsearch::
$ brew link --force elasticsearch@5.6
* If you need to update dependencies::
$ brew update
$ rm -rf encoded/eggs
* If you need to upgrade brew-installed packages that don't have pinned versions,
you can use the following. However, take care because there is no command to directly
undo this effect::
$ brew update
$ brew upgrade
$ rm -rf encoded/eggs
Step 3: Running Poetry
----------------------
To locally install using versions of Python libraries that have worked before, use this::
$ poetry install
Updating dependencies
=====================
To update the version dependencies, use::
$ poetry update
This command also takes space-separated names of specific packages to update. For more information, do::
$ poetry help update
Managing poetry.lock after update
---------------------------------
There may be situations where you do this with no intent to check in the resulting updates,
but once you have checked that the updates are sound, you may wish to check the resulting
``poetry.lock`` file.
Publishing
==========
Normally, a successful build on a tagged branch (including a branch tagged as a beta)
will cause publication automatically. The process begins by obtaining the version. You might do
$ head pyproject.toml
to see the first few lines of `pyproject.toml`, which will contain a line like ``version = 100.200.300``, which
is the ``snovault`` version. You should prepend the letter ``v`` to that version, and create the tag and push
it to the GitHub server:
$ git tag v100.200.300
$ git push origin v100.200.300
Please do NOT use some other syntax for ``git push`` that pushes all of your tags. That might pick up tags that
do not belong on the server and can generally cause tag pollution. Push specifically the tag you intend to publish.
Pushing such a tag should trigger publication automatically within a few minutes.
Manual Publication
------------------
There might be rare occasions where you need to do the publication manually, but normally it is not necessary
or desirable. In most cases, it will either fail or will cause the automatic publication step to fail. The main
case where this is known to be needed is where publication has failed on a tagged branch for reasons other than
the fact of that tag being already published (e.g., a network interruption or a premature shutdown of the GitHub
Actions task). An incomplete publication on GitHub Actions cannot be easily retried, so only in that case you may
need to do:
$ make publish
However, to do this command locally, you would need appropriate credentials on PyPi for such publication to succeed.
As presently configured, these credentials need to be in the environment variables ``PYPI_USER`` and ``PYPI_PASSWORD``.
The script that runs if you manually attempt ``make publish`` checks that you have properly declared credentials
before it attempts to publish. Note that GitHub Actions is already configured with credentials, so you do not
need to worry about them if you just push a tag and let the pre-defined action do the publication.
Running tests
=============
To run specific tests locally::
$ bin/test -k test_name
To run with a debugger::
$ bin/test --pdb
Specific tests to run locally for schema changes::
$ bin/test -k test_load_workbook
Run the Pyramid tests with::
$ bin/test
Raw data
{
"_id": null,
"home_page": "https://github.com/4dn-dcic/snovault",
"name": "dcicsnovault",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.9",
"maintainer_email": null,
"keywords": null,
"author": "4DN-DCIC Team",
"author_email": "support@4dnucleome.org",
"download_url": "https://files.pythonhosted.org/packages/04/c1/8c28b863e3049933346b85cc35d335ea142d3ecb68202a03b8c01890fec7/dcicsnovault-11.23.0.tar.gz",
"platform": null,
"description": "=============\nDCIC Snovault\n=============\n\n|Build status|_\n\n.. |Build status| image:: https://travis-ci.org/4dn-dcic/snovault.svg?branch=master\n.. _Build status: https://travis-ci.org/4dn-dcic/snovault\n\n.. Important::\n\n DCIC Snovault is a FORK of `snovault <https://pypi.org/project/snovault/>`_\n created at the `ENCODE DCC project at Stanford <https://github.com/ENCODE-DCC>`_.\n Our fork supports other projects of the\n `4D Nucleome Data Coordination and Integration Center (4DN-DCIC)\n <https://github.com/4dn-dcic>`_.\n Although this software is available as open source software,\n its primary function is to support our layered projects,\n and we are not at this time able to offer any active support for other uses.\n In particular, this fork does not purport to supersede\n the original `snovault <https://pypi.org/project/snovault/>`_.\n we just have a different use case that we are actively exploring.\n\nOverview\n========\n\nDCIC Snovault is a JSON-LD Database Framework that serves as the backend for the 4DN Data portal and CGAP. Check out our full documentation `here\n<https://snovault.readthedocs.io/en/latest/>`_.\n\n.. note::\n\n This repository contains a core piece of functionality shared amongst several projects\n in the 4DN-DCIC. It is meant to be used internally by the DCIC team\n in support of `Fourfront <https://data.4dnucleome.org>`_\\ ,\n the 4DN data portal, and at this point in time it is not expected to be useful\n in a standalone/plug-and-play way to others.\n\nInstallation in 4DN components\n==============================\n\nDCIC Snovault is pip installable as the ``dcicsnovault`` package with::\n\n $ pip install dcicsnovault``\n\nHowever, at the present time, the functionality it provides might only be useful in conjunction\nwith other 4DN-DCIC components.\n\nNOTE: If you'd like to enable Elasticsearch mapping with type=nested, set the environment variable \"MAPPINGS_USE_NESTED\"\nor set the registry setting \"mappings.use_nested\".\n\nInstallation for Development\n============================\n\nCurrently these are for Mac OSX using homebrew. If using linux, install dependencies with a different package manager.\n\nStep 0: Install Xcode\n---------------------\n\nInstall Xcode (from App Store) and homebrew: http://brew.sh\n\nStep 1: Verify Homebrew Itself\n------------------------------\n\nVerify that homebrew is working properly::\n\n $ brew doctor\n\nStep 2: Install Homebrewed Dependencies\n---------------------------------------\n\nInstall or update dependencies::\n\n $ brew install libevent libmagic libxml2 libxslt openssl postgresql graphviz python3\n $ brew install freetype libjpeg libtiff littlecms webp # Required by Pillow\n $ brew cask install adoptopenjdk8\n $ brew install elasticsearch@5.6\n\nNOTES:\n\n* If installation of adtopopenjdk8 fails due to an ambiguity, it should work to do this instead::\n\n $ brew cask install homebrew/cask-versions/adoptopenjdk8\n\n* If you try to invoke elasticsearch and it is not found,\n you may need to link the brew-installed elasticsearch::\n\n $ brew link --force elasticsearch@5.6\n\n* If you need to update dependencies::\n\n $ brew update\n $ rm -rf encoded/eggs\n\n* If you need to upgrade brew-installed packages that don't have pinned versions,\n you can use the following. However, take care because there is no command to directly\n undo this effect::\n\n $ brew update\n $ brew upgrade\n $ rm -rf encoded/eggs\n\nStep 3: Running Poetry\n----------------------\n\nTo locally install using versions of Python libraries that have worked before, use this::\n\n $ poetry install\n\n\nUpdating dependencies\n=====================\n\nTo update the version dependencies, use::\n\n $ poetry update\n\nThis command also takes space-separated names of specific packages to update. For more information, do::\n\n $ poetry help update\n\n\nManaging poetry.lock after update\n---------------------------------\n\nThere may be situations where you do this with no intent to check in the resulting updates,\nbut once you have checked that the updates are sound, you may wish to check the resulting\n``poetry.lock`` file.\n\nPublishing\n==========\n\nNormally, a successful build on a tagged branch (including a branch tagged as a beta)\nwill cause publication automatically. The process begins by obtaining the version. You might do\n\n $ head pyproject.toml\n\nto see the first few lines of `pyproject.toml`, which will contain a line like ``version = 100.200.300``, which\nis the ``snovault`` version. You should prepend the letter ``v`` to that version, and create the tag and push\nit to the GitHub server:\n\n $ git tag v100.200.300\n $ git push origin v100.200.300\n\nPlease do NOT use some other syntax for ``git push`` that pushes all of your tags. That might pick up tags that\ndo not belong on the server and can generally cause tag pollution. Push specifically the tag you intend to publish.\n\nPushing such a tag should trigger publication automatically within a few minutes.\n\nManual Publication\n------------------\n\nThere might be rare occasions where you need to do the publication manually, but normally it is not necessary\nor desirable. In most cases, it will either fail or will cause the automatic publication step to fail. The main\ncase where this is known to be needed is where publication has failed on a tagged branch for reasons other than\nthe fact of that tag being already published (e.g., a network interruption or a premature shutdown of the GitHub\nActions task). An incomplete publication on GitHub Actions cannot be easily retried, so only in that case you may\nneed to do:\n\n $ make publish\n\nHowever, to do this command locally, you would need appropriate credentials on PyPi for such publication to succeed.\nAs presently configured, these credentials need to be in the environment variables ``PYPI_USER`` and ``PYPI_PASSWORD``.\nThe script that runs if you manually attempt ``make publish`` checks that you have properly declared credentials\nbefore it attempts to publish. Note that GitHub Actions is already configured with credentials, so you do not\nneed to worry about them if you just push a tag and let the pre-defined action do the publication.\n\nRunning tests\n=============\n\nTo run specific tests locally::\n\n $ bin/test -k test_name\n\nTo run with a debugger::\n\n $ bin/test --pdb\n\nSpecific tests to run locally for schema changes::\n\n $ bin/test -k test_load_workbook\n\nRun the Pyramid tests with::\n\n $ bin/test\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Storage support for 4DN Data Portals.",
"version": "11.23.0",
"project_urls": {
"Documentation": "https://github.com/4dn-dcic/snovault",
"Homepage": "https://github.com/4dn-dcic/snovault",
"Repository": "https://github.com/4dn-dcic/snovault"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d9ca12cb17a7b87b381e2bc5f73745f7458b14aa3d862b41fdd6bccbb727c134",
"md5": "42c4122ad5307cd23e09953a333def24",
"sha256": "5ab18fa0c5427fdeb83d3944cc11ee49234e986c8836845c54b79493a6077b65"
},
"downloads": -1,
"filename": "dcicsnovault-11.23.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "42c4122ad5307cd23e09953a333def24",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.9",
"size": 467787,
"upload_time": "2024-11-07T18:25:06",
"upload_time_iso_8601": "2024-11-07T18:25:06.324831Z",
"url": "https://files.pythonhosted.org/packages/d9/ca/12cb17a7b87b381e2bc5f73745f7458b14aa3d862b41fdd6bccbb727c134/dcicsnovault-11.23.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "04c18c28b863e3049933346b85cc35d335ea142d3ecb68202a03b8c01890fec7",
"md5": "72422d7bb0ca5eefacdd8996e5092c61",
"sha256": "44f1169d2d59b1b0015cbde116b9b21e98ba4cca42f905b9ced3a544d10ba423"
},
"downloads": -1,
"filename": "dcicsnovault-11.23.0.tar.gz",
"has_sig": false,
"md5_digest": "72422d7bb0ca5eefacdd8996e5092c61",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.9",
"size": 394450,
"upload_time": "2024-11-07T18:25:09",
"upload_time_iso_8601": "2024-11-07T18:25:09.024048Z",
"url": "https://files.pythonhosted.org/packages/04/c1/8c28b863e3049933346b85cc35d335ea142d3ecb68202a03b8c01890fec7/dcicsnovault-11.23.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-07 18:25:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "4dn-dcic",
"github_project": "snovault",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dcicsnovault"
}
JSON
