ckanext-dcor_depot
==================
|PyPI Version| |Build Status| |Coverage Status|
This plugin manages how data are stored in DCOR. There are two types of
files in DCOR:
1. Resources uploaded by users, imported from figshare, or
imported from a data archive
2. Ancillary files that are generated upon resource creation, such as
condensed DC data, preview images (see
`ckanext-dc_view <https://github.com/DCOR-dev/ckanext-dc_view>`_).
This plugin implements:
- A background job that uploads resources to S3 in `after_resource_create`
if the resources were previously uploaded via the legacy upload route.
This functionality is interesting for legacy setups that are not using
S3 object storage by default.
- Import datasets from figshare. Existing datasets on figshare are
parsed, datasets are created and resources uploaded to S3 object storage.
When running the following command, the "figshare-import" organization
is created and the datasets listed in ``figshare_dois.txt`` are added to CKAN:
::
ckan dcor-import-figshare --limit 2
- CLI for running all background jobs (migration to S3):
::
ckan run-jobs-dcor-depot
- CLI for appending a resource to a dataset
::
ckan append-resource /path/to/file dataset_id --delete-source
- CLI for migrating data from block storage to an S3-compatible object storage
service. For this, the following configuration keys must be specified in
the ``ckan.ini`` file::
dcor_object_store.access_key_id = ACCESS_KEY_ID
dcor_object_store.secret_access_key = SECRET_ACCESS_KEY
dcor_object_store.endpoint_url = S3_ENDPOINT_URL
dcor_object_store.ssl_verify = true
# The bucket name is by default defined by the circle ID. Resources
# are stored in the "RES/OUR/CEID-SCHEME" in that bucket.
dcor_object_store.bucket_name = circle-{organization_id}
Usage::
ckan dcor-migrate-resources-to-object-store --modified-days 2 --delete-after-migration --verify-checksum
- CLI for listing all S3 objects for a dataset::
ckan dcor-list-s3-objects-for-dataset c7a98a04-4e0a-98a7-fb0b-eca379d1f219
- CLI for listing all resources::
ckan list-all-resources
- CLI for pruning stale multipart uploads::
ckan dcor-prune-stale-multipart-uploads --initiated-before-days 5 --dry-run
Installation
------------
::
pip install ckanext-dcor_depot
Add this extension to the plugins and default_views in ckan.ini:
::
ckan.plugins = [...] dcor_depot
ckan.storage_path=/data/ckan-HOSTNAME
ckanext.dcor_depot.users_depot_name=users-HOSTNAME
This plugin stores resources to `/data`:
::
mkdir -p /data/depots/users-$(hostname)
chown -R www-data /data/depots/users-$(hostname)
Testing
-------
If CKAN/DCOR is installed and setup for testing, this extension can
be tested with pytest:
::
pytest ckanext
Testing is implemented via GitHub Actions. You may also set up a local
docker container with CKAN and MinIO. Take a look at the GitHub Actions
workflow for more information.
.. |PyPI Version| image:: https://img.shields.io/pypi/v/ckanext.dcor_depot.svg
:target: https://pypi.python.org/pypi/ckanext.dcor_depot
.. |Build Status| image:: https://img.shields.io/github/actions/workflow/status/DCOR-dev/ckanext-dcor_depot/check.yml
:target: https://github.com/DCOR-dev/ckanext-dcor_depot/actions?query=workflow%3AChecks
.. |Coverage Status| image:: https://img.shields.io/codecov/c/github/DCOR-dev/ckanext-dcor_depot
:target: https://codecov.io/gh/DCOR-dev/ckanext-dcor_depot
Raw data
{
"_id": null,
"home_page": null,
"name": "ckanext-dcor-depot",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.8",
"maintainer_email": "Paul M\u00fcller <dev@craban.de>",
"keywords": "DC, DCOR, deformability, cytometry",
"author": "Paul M\u00fcller",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/e7/df/1377580d8fb9bc137a811bc632e8fa872c00adef361075d932a4582efb94/ckanext_dcor_depot-1.0.3.tar.gz",
"platform": null,
"description": "ckanext-dcor_depot\n==================\n\n|PyPI Version| |Build Status| |Coverage Status|\n\nThis plugin manages how data are stored in DCOR. There are two types of\nfiles in DCOR:\n\n1. Resources uploaded by users, imported from figshare, or\n imported from a data archive\n2. Ancillary files that are generated upon resource creation, such as\n condensed DC data, preview images (see\n `ckanext-dc_view <https://github.com/DCOR-dev/ckanext-dc_view>`_).\n\nThis plugin implements:\n\n- A background job that uploads resources to S3 in `after_resource_create`\n if the resources were previously uploaded via the legacy upload route.\n This functionality is interesting for legacy setups that are not using\n S3 object storage by default.\n- Import datasets from figshare. Existing datasets on figshare are\n parsed, datasets are created and resources uploaded to S3 object storage.\n When running the following command, the \"figshare-import\" organization\n is created and the datasets listed in ``figshare_dois.txt`` are added to CKAN:\n\n ::\n\n ckan dcor-import-figshare --limit 2\n\n\n- CLI for running all background jobs (migration to S3):\n\n ::\n\n ckan run-jobs-dcor-depot\n\n\n- CLI for appending a resource to a dataset\n\n ::\n\n ckan append-resource /path/to/file dataset_id --delete-source\n\n\n\n- CLI for migrating data from block storage to an S3-compatible object storage\n service. For this, the following configuration keys must be specified in\n the ``ckan.ini`` file::\n\n dcor_object_store.access_key_id = ACCESS_KEY_ID\n dcor_object_store.secret_access_key = SECRET_ACCESS_KEY\n dcor_object_store.endpoint_url = S3_ENDPOINT_URL\n dcor_object_store.ssl_verify = true\n # The bucket name is by default defined by the circle ID. Resources\n # are stored in the \"RES/OUR/CEID-SCHEME\" in that bucket.\n dcor_object_store.bucket_name = circle-{organization_id}\n\n Usage::\n\n ckan dcor-migrate-resources-to-object-store --modified-days 2 --delete-after-migration --verify-checksum\n\n- CLI for listing all S3 objects for a dataset::\n\n ckan dcor-list-s3-objects-for-dataset c7a98a04-4e0a-98a7-fb0b-eca379d1f219\n\n- CLI for listing all resources::\n\n ckan list-all-resources\n\n- CLI for pruning stale multipart uploads::\n\n ckan dcor-prune-stale-multipart-uploads --initiated-before-days 5 --dry-run\n\n\nInstallation\n------------\n\n::\n\n pip install ckanext-dcor_depot\n\n\nAdd this extension to the plugins and default_views in ckan.ini:\n\n::\n\n ckan.plugins = [...] dcor_depot\n ckan.storage_path=/data/ckan-HOSTNAME\n ckanext.dcor_depot.users_depot_name=users-HOSTNAME\n\nThis plugin stores resources to `/data`:\n\n::\n\n mkdir -p /data/depots/users-$(hostname)\n chown -R www-data /data/depots/users-$(hostname)\n\n\nTesting\n-------\nIf CKAN/DCOR is installed and setup for testing, this extension can\nbe tested with pytest:\n\n::\n\n pytest ckanext\n\nTesting is implemented via GitHub Actions. You may also set up a local\ndocker container with CKAN and MinIO. Take a look at the GitHub Actions\nworkflow for more information.\n\n\n.. |PyPI Version| image:: https://img.shields.io/pypi/v/ckanext.dcor_depot.svg\n :target: https://pypi.python.org/pypi/ckanext.dcor_depot\n.. |Build Status| image:: https://img.shields.io/github/actions/workflow/status/DCOR-dev/ckanext-dcor_depot/check.yml\n :target: https://github.com/DCOR-dev/ckanext-dcor_depot/actions?query=workflow%3AChecks\n.. |Coverage Status| image:: https://img.shields.io/codecov/c/github/DCOR-dev/ckanext-dcor_depot\n :target: https://codecov.io/gh/DCOR-dev/ckanext-dcor_depot\n",
"bugtrack_url": null,
"license": "GNU Affero General Public License v3 or later (AGPLv3+)",
"summary": "Manages data storage for DCOR",
"version": "1.0.3",
"project_urls": {
"changelog": "https://github.com/DCOR-dev/ckanext-dcor_depot/blob/main/CHANGELOG",
"source": "https://github.com/DCOR-dev/ckanext-dcor_depot",
"tracker": "https://github.com/DCOR-dev/ckanext-dcor_depot/issues"
},
"split_keywords": [
"dc",
" dcor",
" deformability",
" cytometry"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "36ebe8aaecb31e196385736abec464d31efde82ece8a85fa309adaf21566fa7a",
"md5": "bf80506bea92fb640f9e22894fc3de92",
"sha256": "5511fcc2aa87403d9ac2d8d407a58d88ca53d7a719b25e6c27e85a614536863f"
},
"downloads": -1,
"filename": "ckanext_dcor_depot-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bf80506bea92fb640f9e22894fc3de92",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.8",
"size": 4294987,
"upload_time": "2025-07-12T23:22:12",
"upload_time_iso_8601": "2025-07-12T23:22:12.116285Z",
"url": "https://files.pythonhosted.org/packages/36/eb/e8aaecb31e196385736abec464d31efde82ece8a85fa309adaf21566fa7a/ckanext_dcor_depot-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e7df1377580d8fb9bc137a811bc632e8fa872c00adef361075d932a4582efb94",
"md5": "83f6272ad37019dc2e7ccc1578bcfd1d",
"sha256": "975ba03b5f10d8d528d764a7a7f3c7d9650c7edce259527506bca2ad4faef307"
},
"downloads": -1,
"filename": "ckanext_dcor_depot-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "83f6272ad37019dc2e7ccc1578bcfd1d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.8",
"size": 4297815,
"upload_time": "2025-07-12T23:22:13",
"upload_time_iso_8601": "2025-07-12T23:22:13.509381Z",
"url": "https://files.pythonhosted.org/packages/e7/df/1377580d8fb9bc137a811bc632e8fa872c00adef361075d932a4582efb94/ckanext_dcor_depot-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-12 23:22:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DCOR-dev",
"github_project": "ckanext-dcor_depot",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "ckanext-dcor-depot"
}
JSON
