# BDRC-UTIL
## Overview
BDRC UTIL is a python package containing modules for use by the Buddhist Digital Resource Center. It is offered to
the public under the [MIT License](https://mit-license.org). This document describes its contents and features.
Although publicly available, BDRC does not support this project for use by others. We will not respond to
questions about its features and functionality.
## Development
archive-ops uses python packages from `archive-ops/venv`
### Deployment
```shell
# be in project main dir
python -m setup bdist_wheel
# test
twine upload --verbose -r testpypi dist/bdrc_util-x.MM.mm-py3-none-any.whl
# prod
twine upload --verbose dist/bdrc_util-x.MM.mm-py3-none-any.whl
```
## Installation
[pyPI.org bdrc-util](https://pypi.org/project/bdrc-util/)
### Debian requirements
You need this (and its dependencies) for the pip
component mysqlclient to install
`sudo apt install default-libmysqlclient-dev`
### MacOS requirements
You need this (and its dependencies) for the pip
component mysqlclient to install
`brew install mysql`
## Contents
### Publicly available scripts
As defined in setup.py
### locators
Maps a work and a destination parent to a specific directory using various BDRC mapping schemes
### migrate works
Scripts to migrate and log works into BDRC's 2021 Archival strategy
### log_dip
Log creation and distribution of Distribution Information Packages (DIPs). DIP is
an OAIS term to describe a unit of publication.
## User Guides
### `log_dip`
The command `log_dip` is intended for use by BDRC staff to instrument their publication activities.
`log_dip` takes arguments from the shell and transfers them into a database table.
#### Synopsis
```
log_dip --help
usage: DipLog.py | -d DBAppSection:DbAppFile DipLog.py
Logs a number of different publication strategies
positional arguments:
dip_source_path Source path - string, (optional)
dip_dest_path Destination path (optional) - string
optional arguments:
-h, --help show this help message and exit
-d DRSDBCONFIG, --drsDbConfig DRSDBCONFIG
specify section:configFileName
-l {info,warning,error,debug,critical}, --log-level {info,warning,error,debug,critical}
choice values are from python logging module
-a {DRS,IA,BUDA,DEEP_ARCHIVE,ARCHIVE}, --activity_type {DRS,IA,BUDA,DEEP_ARCHIVE,ARCHIVE}
Destination repository
-w WORK_NAME, --work_name WORK_NAME
work being distributed
-i DIP_ID, --dip_id DIP_ID
ID to update
-r ACTIVITY_RETURN_CODE, --activity_return_code ACTIVITY_RETURN_CODE
Integer result of operation.
-b BEGIN_TIME, --begin_time BEGIN_TIME
time of beginning - ')yyyy-mm-dd hh:mm:ss bash format
date +'%Y-%m-%d %R:%S'
-e END_TIME, --end_time END_TIME
time of end.Default is invocation time. yyyy-mm-dd
hh:mm:ss bash format date + '%Y-%m-%d %R:%S'
-c COMMENT, --comment COMMENT
Any text up to 4GB in length
```
#### Argument structure
log_dip creates a database record that captures the beginning or end of a DIP event.
All its operations return an opaque identifier which can reference the record. In bash, this would
be invoked as
You reference the record later by one of two methods:
* passing in the id from the initial (or subsequent calls):
```shell
dip_id=$(dip_log --drsDbConfig sec:some.config --begin_time "2021-05-11 01:23:45" --activity_type DRS --work_name W12345)
dip_log -d sec:some.config --activity_return_code 42 --end_time "2021-05-11 12:34:56" --dip_id $dip_id
```
* using the work Id, Activity type and begin time:
```shell
dip_log -d sec:some.config -b "2021-05-11 01:23:45" -a DRS -w W12345
dip_log -d sec:some.config -b "2021-05-11 01:23:45" -a DRS -w W12345 -r 42 -e "2021-05-11 12:34:56"
```
Both of the above examples perform the same function:
1. log the start of a DRS job for work W12345 at "2021-05-11 01:23:45"
1. log the end_time of the job at "2021-05-11 12:34:56" , with a return code of 42
#### Argument hints
* to give an end time, you must give all the job id information, either in the id, or with the (work_name, begin_time,
activity_id) tuple
* You can add as much information as you want in one call. If you've captured the begin time, you can
create a call which logs them all at the same time (this is not the best practice, because it eliminates the system's
ability to check for in-progress jobs). This is perfectly legal:
```shell
dip_log -d sec:some.config -b "2021-05-11 01:23:45" -a DRS -w W12345 -r 42 -e "2021-05-11 12:34:56 -c "Hi Mom, Im re-writing history"
```
* Begin and end dates are fussy: in shell, the format for generating the date dip_log requires is:
`date +%Y-%m-%d %R:%S` (for Mac with GNU core, GNU Linuxes)
* you can update some DIP log properties:
- comments
- end time
- operation return code
* Obviously, since these are the tuple which identifies the transaction, you cannot modify:
- work name
- begin time
- activity type
In this example, the comments field is updated.
```shell
dip_log_id=$( dip_log -d sec:some.config -b "2021-05-11 01:23:45" -a DRS -w W12345 -r 42 -e "2021-05-11 12:34:56 -c "Experienced some discomfort")
dip_log -d sec:some.config -i $dip_log_id -c "But it passed.")
```
*
* Any property not given in the command line is preserved. (The example above preserves the begin and end
times of the DIP transaction.)
* the comment field is a free-form text field of up to 4GB in length. You can store XML or JSON data in it for later
use. (such as any error messages
or summary information about the process or the objects being processed)
# API
A simple API, inspired by `openpecha.buda.api` is provided as a central library for commonly used utilities, including
Legacy Hack
Image Group Translation
To use in your code, `pip install bdrc-util>=0.9.44`
# bdrc-util Changelog
| version | commit | Comments |
|---------|---------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|
| 1.0.1 | [ccd9865](https://github.com/buda-base/archive-ops/pull/1068/commits/ccd9865df60877b5bbaf8e442db627af8a5fa652) | `dip_log` passes db config to ORM |
| 0.9.48 | [9573f3c](https://github.com/buda-base/archive-ops/commit/9573f3c7874af0e3e35b88143248898516a7a463) | optional symlink resolution |
| 0.9.47 | [192c43f4](https://github.com/buda-base/archive-ops/commit/b7dd40c9a928007bfda076ec4ff5a6e48cde0c99) | Add s3pathlib to install requirements |
| 0.9.46 | [e14b3a6](https://github.com/buda-base/archive-ops/commit/e14b3a64c5a2618e261aba074403057f18479f51) | decomission web in favor of api |
| 0.9.45 | [89724ee](https://github.com/buda-base/archive-ops/commit/89724eec20b1861f1611fadbe1d46a9c7df14025) | Raise pageSize for Get volumes |
| 0.9.44 | [TBD](https://github.com/buda-base/archive-ops/commit/013242a8a05c47173d94156f13d881364eea06ce) | Move Resolvers to api |
| 0.9.43 | [013242a](https://github.com/buda-base/archive-ops/commit/013242a8a05c47173d94156f13d881364eea06ce) | cacheing to reduce load on server |
| 0.9.42 | [146bc43a](https://github.com/buda-base/archive-ops/commit/146bc43ac4b1e6938398b44c1fcb682cdfe1aba1) | support buda-dld |
| 0.9.41 | [0d01394](https://github.com/buda-base/archive-ops/commit/0d0139448d80b2900f2212b99c40c973ef060532) | print, dont return from `disk_ig_from_buda` |
| 0.9.40 | | Rename get_image_groups |
| 0.9.39 | | Added measure archive fixity |
| | | Shorten log file name |
| 0.9.38 | | Added RST documentation to setup. |
| | | Added minimum requirement for bdrc-db-lib |
| 0.9.34 | | Use external address for resolver |
| 0.9.32 | be754999 | Create entry points for image group renaming |
| 0.9.31 | 192eea17 | (not released) single entry point for image group renames |
| 0.9.30 | 83c5062a | Add Work calculation size to script |
Raw data
{
"_id": null,
"home_page": "https://github.com/buda-base/archive-ops/tree/master/bdrc-util",
"name": "bdrc-util",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "jimk",
"author_email": "jimk@bdrc.io",
"download_url": "",
"platform": null,
"description": "# BDRC-UTIL\n\n## Overview\n\nBDRC UTIL is a python package containing modules for use by the Buddhist Digital Resource Center. It is offered to\nthe public under the [MIT License](https://mit-license.org). This document describes its contents and features.\n\nAlthough publicly available, BDRC does not support this project for use by others. We will not respond to\nquestions about its features and functionality.\n\n## Development\n\narchive-ops uses python packages from `archive-ops/venv`\n\n### Deployment\n\n```shell\n# be in project main dir\npython -m setup bdist_wheel\n# test\ntwine upload --verbose -r testpypi dist/bdrc_util-x.MM.mm-py3-none-any.whl\n# prod \ntwine upload --verbose dist/bdrc_util-x.MM.mm-py3-none-any.whl \n```\n\n## Installation\n\n[pyPI.org bdrc-util](https://pypi.org/project/bdrc-util/)\n\n### Debian requirements\n\nYou need this (and its dependencies) for the pip\ncomponent mysqlclient to install\n`sudo apt install default-libmysqlclient-dev`\n\n### MacOS requirements\n\nYou need this (and its dependencies) for the pip\ncomponent mysqlclient to install\n`brew install mysql`\n\n## Contents\n\n### Publicly available scripts\n\nAs defined in setup.py\n\n### locators\n\nMaps a work and a destination parent to a specific directory using various BDRC mapping schemes\n\n### migrate works\n\nScripts to migrate and log works into BDRC's 2021 Archival strategy\n\n### log_dip\n\nLog creation and distribution of Distribution Information Packages (DIPs). DIP is\nan OAIS term to describe a unit of publication.\n\n## User Guides\n\n### `log_dip`\n\nThe command `log_dip` is intended for use by BDRC staff to instrument their publication activities.\n`log_dip` takes arguments from the shell and transfers them into a database table.\n\n#### Synopsis\n\n```\nlog_dip --help\nusage: DipLog.py | -d DBAppSection:DbAppFile DipLog.py \n\nLogs a number of different publication strategies\n\npositional arguments:\n dip_source_path Source path - string, (optional)\n dip_dest_path Destination path (optional) - string\n\noptional arguments:\n -h, --help show this help message and exit\n -d DRSDBCONFIG, --drsDbConfig DRSDBCONFIG\n specify section:configFileName\n -l {info,warning,error,debug,critical}, --log-level {info,warning,error,debug,critical}\n choice values are from python logging module\n -a {DRS,IA,BUDA,DEEP_ARCHIVE,ARCHIVE}, --activity_type {DRS,IA,BUDA,DEEP_ARCHIVE,ARCHIVE}\n Destination repository\n -w WORK_NAME, --work_name WORK_NAME\n work being distributed\n -i DIP_ID, --dip_id DIP_ID\n ID to update\n -r ACTIVITY_RETURN_CODE, --activity_return_code ACTIVITY_RETURN_CODE\n Integer result of operation.\n -b BEGIN_TIME, --begin_time BEGIN_TIME\n time of beginning - ')yyyy-mm-dd hh:mm:ss bash format\n date +'%Y-%m-%d %R:%S'\n -e END_TIME, --end_time END_TIME\n time of end.Default is invocation time. yyyy-mm-dd\n hh:mm:ss bash format date + '%Y-%m-%d %R:%S'\n -c COMMENT, --comment COMMENT\n Any text up to 4GB in length\n```\n\n#### Argument structure\n\nlog_dip creates a database record that captures the beginning or end of a DIP event.\n\nAll its operations return an opaque identifier which can reference the record. In bash, this would\nbe invoked as\n\nYou reference the record later by one of two methods:\n\n* passing in the id from the initial (or subsequent calls):\n\n```shell\ndip_id=$(dip_log --drsDbConfig sec:some.config --begin_time \"2021-05-11 01:23:45\" --activity_type DRS --work_name W12345)\n\ndip_log -d sec:some.config --activity_return_code 42 --end_time \"2021-05-11 12:34:56\" --dip_id $dip_id\n```\n\n* using the work Id, Activity type and begin time:\n\n```shell\ndip_log -d sec:some.config -b \"2021-05-11 01:23:45\" -a DRS -w W12345\n\ndip_log -d sec:some.config -b \"2021-05-11 01:23:45\" -a DRS -w W12345 -r 42 -e \"2021-05-11 12:34:56\"\n```\n\nBoth of the above examples perform the same function:\n\n1. log the start of a DRS job for work W12345 at \"2021-05-11 01:23:45\"\n1. log the end_time of the job at \"2021-05-11 12:34:56\" , with a return code of 42\n\n#### Argument hints\n\n* to give an end time, you must give all the job id information, either in the id, or with the (work_name, begin_time,\n activity_id) tuple\n* You can add as much information as you want in one call. If you've captured the begin time, you can\n create a call which logs them all at the same time (this is not the best practice, because it eliminates the system's\n ability to check for in-progress jobs). This is perfectly legal:\n\n ```shell\n dip_log -d sec:some.config -b \"2021-05-11 01:23:45\" -a DRS -w W12345 -r 42 -e \"2021-05-11 12:34:56 -c \"Hi Mom, Im re-writing history\"\n``` \n\n* Begin and end dates are fussy: in shell, the format for generating the date dip_log requires is:\n `date +%Y-%m-%d %R:%S` (for Mac with GNU core, GNU Linuxes)\n\n* you can update some DIP log properties:\n - comments\n - end time\n - operation return code\n\n* Obviously, since these are the tuple which identifies the transaction, you cannot modify:\n - work name\n - begin time\n - activity type\n\nIn this example, the comments field is updated.\n\n```shell\ndip_log_id=$( dip_log -d sec:some.config -b \"2021-05-11 01:23:45\" -a DRS -w W12345 -r 42 -e \"2021-05-11 12:34:56 -c \"Experienced some discomfort\")\ndip_log -d sec:some.config -i $dip_log_id -c \"But it passed.\")\n``` \n\n*\n* Any property not given in the command line is preserved. (The example above preserves the begin and end\n times of the DIP transaction.)\n\n* the comment field is a free-form text field of up to 4GB in length. You can store XML or JSON data in it for later\n use. (such as any error messages\n or summary information about the process or the objects being processed)\n\n# API\n\nA simple API, inspired by `openpecha.buda.api` is provided as a central library for commonly used utilities, including\nLegacy Hack\nImage Group Translation\n\nTo use in your code, `pip install bdrc-util>=0.9.44`\n\n# bdrc-util Changelog\n\n| version | commit | Comments |\n|---------|---------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|\n| 1.0.1 | [ccd9865](https://github.com/buda-base/archive-ops/pull/1068/commits/ccd9865df60877b5bbaf8e442db627af8a5fa652) | `dip_log` passes db config to ORM |\n| 0.9.48 | [9573f3c](https://github.com/buda-base/archive-ops/commit/9573f3c7874af0e3e35b88143248898516a7a463) | optional symlink resolution |\n| 0.9.47 | [192c43f4](https://github.com/buda-base/archive-ops/commit/b7dd40c9a928007bfda076ec4ff5a6e48cde0c99) | Add s3pathlib to install requirements |\n| 0.9.46 | [e14b3a6](https://github.com/buda-base/archive-ops/commit/e14b3a64c5a2618e261aba074403057f18479f51) | decomission web in favor of api |\n| 0.9.45 | [89724ee](https://github.com/buda-base/archive-ops/commit/89724eec20b1861f1611fadbe1d46a9c7df14025) | Raise pageSize for Get volumes |\n| 0.9.44 | [TBD](https://github.com/buda-base/archive-ops/commit/013242a8a05c47173d94156f13d881364eea06ce) | Move Resolvers to api | \n| 0.9.43 | [013242a](https://github.com/buda-base/archive-ops/commit/013242a8a05c47173d94156f13d881364eea06ce) | cacheing to reduce load on server | \n| 0.9.42 | [146bc43a](https://github.com/buda-base/archive-ops/commit/146bc43ac4b1e6938398b44c1fcb682cdfe1aba1) | support buda-dld | \n| 0.9.41 | [0d01394](https://github.com/buda-base/archive-ops/commit/0d0139448d80b2900f2212b99c40c973ef060532) | print, dont return from `disk_ig_from_buda` | \n| 0.9.40 | | Rename get_image_groups | \n| 0.9.39 | | Added measure archive fixity | \n| | | Shorten log file name | \n| 0.9.38 | | Added RST documentation to setup. |\n| | | Added minimum requirement for bdrc-db-lib | \n| 0.9.34 | | Use external address for resolver |\n| 0.9.32 | be754999 | Create entry points for image group renaming |\n| 0.9.31 | 192eea17 | (not released) single entry point for image group renames | \n| 0.9.30 | 83c5062a | Add Work calculation size to script |\n\n\n\n\n\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "BDRC Utilities",
"version": "1.0.1",
"project_urls": {
"Homepage": "https://github.com/buda-base/archive-ops/tree/master/bdrc-util"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1ff59c369c3dff0e7871fffd903cfd95b3be9f6c5e778f7b557231e65be6c24e",
"md5": "b46be231bdd400b34745d98afd4f8d98",
"sha256": "a96fd4240f26b56d965bc6defbb8d347f2428fdd968c5c2b9e24810695ad323a"
},
"downloads": -1,
"filename": "bdrc_util-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b46be231bdd400b34745d98afd4f8d98",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 44254,
"upload_time": "2024-03-11T19:45:20",
"upload_time_iso_8601": "2024-03-11T19:45:20.257341Z",
"url": "https://files.pythonhosted.org/packages/1f/f5/9c369c3dff0e7871fffd903cfd95b3be9f6c5e778f7b557231e65be6c24e/bdrc_util-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-11 19:45:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "buda-base",
"github_project": "archive-ops",
"github_not_found": true,
"lcname": "bdrc-util"
}
JSON
