<p>
<a href="https://www.linkedin.com/in/alexandergbraun" rel="nofollow noreferrer">
<img src="https://www.gomezaparicio.com/wp-content/uploads/2012/03/linkedin-logo-1-150x150.png"
alt="linkedin" width="30px" height="30px"
>
</a>
<a href="https://github.com/theNewFlesh" rel="nofollow noreferrer">
<img src="https://tadeuzagallo.com/GithubPulse/assets/img/app-icon-github.png"
alt="github" width="30px" height="30px"
>
</a>
<a href="https://pypi.org/user/the-new-flesh" rel="nofollow noreferrer">
<img src="https://cdn.iconscout.com/icon/free/png-256/python-2-226051.png"
alt="pypi" width="30px" height="30px"
>
</a>
<a href="http://vimeo.com/user3965452" rel="nofollow noreferrer">
<img src="https://cdn.iconscout.com/icon/free/png-512/movie-52-151107.png?f=avif&w=512"
alt="vimeo" width="30px" height="30px"
>
</a>
<a href="http://www.alexgbraun.com" rel="nofollow noreferrer">
<img src="https://i.ibb.co/fvyMkpM/logo.png"
alt="alexgbraun" width="30px" height="30px"
>
</a>
</p>
<!-- <img id="logo" src="resources/logo.png" style="max-width: 717px"> -->
[](https://github.com/theNewFlesh/hidebound/blob/master/LICENSE)
[](https://github.com/theNewFlesh/hidebound/blob/master/docker/config/pyproject.toml)
[](https://pypi.org/project/hidebound/)
[](https://pepy.tech/project/hidebound)
# Overview
A MLOps framework for generating ML assets and metadata.
Hidebound is an ephemeral database and asset framework used for generating,
validating and exporting assets to various data stores. Hidebound enables
developers to ingest arbitrary sets of files and output them as content and
generated metadata, which has been validated according to specifications they
define.
Assets are placed into an ingress directory, typically reserved for Hidebound
projects, and then processed by Hidebound. Hidebound extracts metadata from the
files and directories that make each asset according to their name, location and
file properties. This data comprises the entirety of Hidebound's database at any
one time.
See [documentation](https://thenewflesh.github.io/hidebound/) for details.
# Installation for Developers
### Docker
1. Install [docker-desktop](https://docs.docker.com/desktop/)
2. Ensure docker-desktop has at least 4 GB of memory allocated to it.
3. `git clone git@github.com:theNewFlesh/hidebound.git`
4. `cd hidebound`
5. `chmod +x bin/hidebound`
6. `bin/hidebound docker-start`
- If building on a M1 Mac run `export DOCKER_DEFAULT_PLATFORM=linux/amd64` first.
The service should take a few minutes to start up.
Run `bin/hidebound --help` for more help on the command line tool.
### ZSH Setup
1. `bin/hidebound` must be run from this repository's top level directory.
2. Therefore, if using zsh, it is recommended that you paste the following line
in your ~/.zshrc file:
- `alias hidebound="cd [parent dir]/hidebound; bin/hidebound"`
- Replace `[parent dir]` with the parent directory of this repository
3. Consider adding the following line to your ~/.zshrc if you are using a M1 Mac:
- `export DOCKER_DEFAULT_PLATFORM=linux/amd64`
4. Running the `zsh-complete` command will enable tab completions of the cli
commands, in the next shell session.
For example:
- `hidebound [tab]` will show you all the cli options, which you can press
tab to cycle through
- `hidebound docker-[tab]` will show you only the cli options that begin with
"docker-"
# Installation for Production
### Python
`pip install hidebound`
### Docker
1. Install [docker-desktop](https://docs.docker.com/desktop/)
2. `docker pull thenewflesh/hidebound:[version]`
---
# Dataflow
Data begins as files on disk. Hidebound creates a JSON-compatible dict from
their name traits and file traits and then constructs an internal database table
from them, one dict per row. All the rows are then aggregated by asset, and
converted into JSON blobs. Those blobs are then validated according to their
respective specifications. Files from valid assets are then copied or moved into
Hidebound's content directory, according to their same directory structure and
naming. Metadata is written to JSON files inside Hidebound's metadata directory.
Each file's metadata is written as a JSON file in /hidebound/metadata/file, and
each asset's metadata (the aggregate of its file metadata) is written to
/hidebound/metadata/asset. From their exporters, can export the valid
asset data and its accompanying metadata to various locations, like an AWS S3
bucket.
# Workflow
The acronynm to remember for workflows is **CRUDES**: create, read, update,
delete, export and search. Those operations constitue the main functionality
that Hidebound supports.
## *Create Asset*
For example, an asset could be an image sequence, such as a directory full of
PNG files, all of which have a frame number, have 3 (RGB) channels, and are 1024
pixels wide by 1024 pixels tall. Let's call the specification for this type of
asset "spec001". We create an image sequence of a cat running, and we move it
into the Hidebound projects directory.
## *Update*
We call the update function via Hidebound's web app. Hidebound creates
a new database based upon the recursive listing of all the files within said
directory. This database is displayed to us as a table, with one file per row.
If we choose to group by asset in the app, the table will display one asset per
row. Hidebound extracts metadata from each filename (not any directory name) as
well as from the file itself. That metadata is called file_traits. Using only
information derived from filename and file traits, Hidebound determines which
files are grouped together as a single asset and the specification of that
asset. Asset traits are then derived from this set of files (one or more).
Finally, Hidebound validates each asset according to its determined
specification. All of this data is displayed as a table within the web app.
Importantly, all of the errors in filenames, file traits and asset traits are
included.
## *Review Graph*
If we click on the graph tab, we are greeted by a hierarchical graph of all our
assets in our project directory. Our asset is red, meaning it's invalid. Valid
asset's are green, and all other files and directories, including parent
directories, are cyan.
## *Diagnose and Repair*
We flip back to the data tab. Using table within it, we search (via SQL) for our
asset within Hidebound's freshly created database. We see an error in one of the
filenames, conveniently displayed in red text. The descriptor in one orf our
filenames has capital letters in it. This violates Hidebound's naming
convention, and so we get an error. We go and rename the file appropriately and
call update again. Our asset is now valid. The filenames are correct and we can
see in the height and width columns, that it's 1024 by 1024 and the channels
column says it has three.
## *Create*
Next we click the create button. For each valid asset, Hidebound generates file
and asset metadata as JSON files within the hidebound/metadata directory.
Hidebound also copies or moves, depending on the config write mode, valid files
and directories into the hidebound/content directory. Hidebound/content and
hidebound/metadata are both staging directories used for generating a valid
ephemeral database. We now have a hidebound directory that looks like this
(unmentioned assets are collapsed behind the ellipses):
```shell
/tmp/hidebound
├── hidebound_config.yaml
│
├── specifications
│ └── specifications.py
│
├── data
│ ...
│ └── p-cat001
│ └── spec001
│ └── p-cat001_s-spec001_d-running-cat_v001
│ ├── p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0001.png
│ ├── p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0002.png
│ └── p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png
│
├── metadata
├── asset
│ ...
│ └── a9f3727c-cb9b-4eb1-bc84-a6bc3b756cc5.json
│
└── file
...
├── 279873a2-bfd0-4757-abf2-7dc4f771f992.json
├── e50160ae-8678-40b3-b766-ee8311b1f0c9.json
└── ea95bd79-cb8f-4262-8489-efe734c5f65c.json
```
## *Export*
The hidebound directories contain only valid assets. Thus, we are now free to
export this data to various data stores, such as AWS S3, MongoDB, and Girder.
Exporters are are defined within the exporters subpackage. They expect a
populated hidebound directory and use the files and metadata therein to export
hidebound data. Exporter configurations are stored in the hidebound config,
under the "exporters" key. Currently supported exporters include, disk, s3 and
girder. Below we can see the results of an export to Girder in the Girder web
app.
## *Delete*
Once this export process is complete, we may click the delete button. Hidebound
deletes the hidebound/content and hidebound/metdata directories and all their
contents. If write_mode in the Hidebound configuration is set to "copy", then
this step will merely delete data created by Hidebound. If it is set to "move",
then Hidebound will presumably delete, the only existing copy of out asset data
on the host machine. The delete stage in combination with the removal of assets
from the ingress directory is what makes Hidebound's database ephemeral.
## *Workflow*
`/api/workflow` is a API endpoint that initializes a database a with a given
config, and then calls each method from a given list. For instance, if you send
this data to `/api/workflow`:
```{config={...}, workflow=['update', 'create', 'export', 'delete']}```
A database instance will be created with the given config, and then that
instance will call its update, create, export and delete methods, in that order.
# Naming Convention
Hidebound is a highly opinionated framework that relies upon a strict but
composable naming convention in order to extract metadata from filenames. All
files and directories that are part of assets must conform to a naming
convention defined within that asset's specification.
In an over-simplified sense; sentences are constructions of words. Syntax
concerns how each word is formed, grammar concerns how to form words into a
sentence, and semantics concerns what each word means. Similarly, filenames can
be thought of as crude sentences. They are made of several words (ie fields).
These words have distinct semantics (as determines by field indicators). Each
word is constructed according to a syntax (ie indicator + token). All words are
joined together by spaces (ie underscores) in a particular order as determined
by grammar (as defined in each specification).
## *Syntax*
- Names consist of a series of fields, each separated by a single underscore
“_”, also called a field separator.
- Periods, ".", are the exception to this, as it indicates file extension.
- Legal characters include and only include:
| Name | Characters | Use |
| ---------------- | ---------- | ------------------------- |
| Underscore | _ | only for field separation |
| Period | . | only for file extensions |
| Lowercase letter | a to z | everything |
| Number | 0 to 9 | everything |
| Hyphen | - | token separator |
Fields are comprised of two main parts:
| Name | Use |
| ---------------- | --------------------------------------------------- |
| Field indicator | determines metadata key |
| Field token | a set of 1+ characters that define the field's data |
---
## **Example Diagrams**
In our example filename:
`p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png` the metadata will be:
```json
{
"project": "cat001",
"specification": "spec001",
"descriptor": "running-cat",
"version": 1,
"coordinate": [0, 5],
"frame": 3,
"extension": "png",
}
```
The spec001 specification is derived from the second field of this filename:
```shell
field field
indicator token
| __|__
| | |
p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png
|_______|
|
field
```
| Part | Value |
| ---------------- | ------------------------ |
| Field | s-spec001 |
| Field indicator | s- |
| Field token | spec001 |
| Derived metadata | {specification: spec001} |
## *Special Field Syntax*
- Projects begin with 3 to 10 letters followed by 1 to 4 numbers
- Specifications begin with 3 or 4 letters followed by 3 numbers
- Descriptors begin with a letter or number and may also contain hyphens
- Descriptors may not begin with the words master, final or last
- Versions are triple-padded with zeros and must be greater than 0
- Coordinates may contain up to 3 quadruple-padded numbers, separated by hyphens
- Coordinates are always evaluated in XYZ order. For example: `c0001-0002-0003`
produces `{x: 1, y: 2, z: 3}`.
- Each element of a coordinate may be equal to or greater than zero
- Frames are quadruple-padded and are greater than or equal to 0
- Extensions may only contain upper and lower case letters a to z and numbers 0
to 9
## *Semantics*
Hidebound is highly opionated, especially with regards to its semantics. It
contains exactly seven field types, as indicated by their field indicators.
They are:
| Field | Indicator |
| ------------- | --------- |
| project | p- |
| specification | s- |
| descriptor | d- |
| version | v |
| coordinate | c |
| frame | f |
| extension | . |
## *Grammar*
The grammar is fairly simple:
- Names are comprised of an ordered set of fields drawn from the seven above
- All names must contain the specification field
- All specification must define a field order
- All fields of a name under that specification must occcur in its defined
field order
Its is highly encouraged that fields be defined in the following order:
`project specification descriptor version coordinate frame extension`
The grammatical concept of field order here is one of rough encapsulation:
- Projects contain assets
- Assets are grouped by specification
- A set of assets of the same content is grouped by a descriptor
- That set of assets consists of multiple versions of the same content
- A single asset may broken into chunks, identified by 1, 2 or 3 coordinates
- Each chunk may consist of a series of files seperated by frame number
- Each file has an extension
## *Encouraged Lexical Conventions*
- Specifications end with a triple padded number so that they may be explicitely
versioned. You redefine an asset specification to something slightly
different, by copying its specification class, adding one to its name and
change the class attributes in some way. That way you always maintain
backwards compatibility with legacy assets.
- Descriptors are not a dumping ground for useless terms like wtf, junk, stuff,
wip and test.
- Descriptors should not specify information known at the asset specification
level, such as the project name, the generic content of the asset (ie image,
mask, png, etc).
- Descriptors should not include information that can be known from the
preceding tokens, such as version, frame or extension.
- A descriptor should be applicable to every version of the asset it designates.
- Use of hyphens in descriptors is encouraged.
- When in doubt, hyphenate and put into the descriptor.
---
# Project Structure
Hidebound does not formally define a project structure. It merely stipulates
that assets must exist under some particular root directory. Each asset
specification does define a directory structure for the files that make up that
asset. Assets are divided into 3 types: file, sequence and complex. File defines
an asset that consists of a single file. Sequence is defined to be a single
directory containing one or more files. Complex is for assets that consist of an
arbitrarily complex layout of directories and files.
The following project structure is recommended:
```shell
project
|-- specification
|-- descriptor
|-- asset # either a file or directory of files and directories
|- file
```
## For Example
```shell
/tmp/projects
└── p-cat001
├── s-spec002
│ ├── d-calico-jumping
│ │ └── p-cat001_s-spec002_d-calico-jumping_v001
│ │ ├── p-cat001_s-spec002_d-calico-jumping_v001_f0001.png
│ │ ├── p-cat001_s-spec002_d-calico-jumping_v001_f0002.png
│ │ └── p-cat001_s-spec002_d-calico-jumping_v001_f0003.png
│ │
│ └── d-tabby-playing
│ ├── p-cat001_s-spec002_d-tabby-playing_v001
│ │ ├── p-cat001_s-spec002_d-tabby-playing_v001_f0001.png
│ │ ├── p-cat001_s-spec002_d-tabby-playing_v001_f0002.png
│ │ └── p-cat001_s-spec002_d-tabby-playing_v001_f0003.png
│ │
│ └── p-cat001_s-spec002_d-tabby-playing_v002
│ ├── p-cat001_s-spec002_d-tabby-playing_v002_f0001.png
│ ├── p-cat001_s-spec002_d-tabby-playing_v002_f0002.png
│ └── p-cat001_s-spec002_d-tabby-playing_v002_f0003.png
│
└── spec001
└── p-cat001_s-spec001_d-running-cat_v001
├── p-cat001_s-spec001_d-Running-Cat_v001_c0000-0005_f0002.png
├── p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0001.png
└── p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png
```
# Application
The Hidebound web application has five sections: data, graph, config, api and
docs.
## Data
The data tab is the workhorse of the Hidebound app.
Its functions are as follows:
* Search - Search the updated database's data via SQL
* Dropdown - Groups search results by file or asset
* Init - Initialized the database with the current config
* Update - Initializes and updates the database with the current config
* Create - Copies or moves valid assets to hidebound/content directory and
creates JSON files in hidebound/metadata directory
* Delete - Deletes hidebound/content and hidebound/metadata directories
Prior to calling update, the application will look like this:
## Graph
The graph tab is used for visualizing the state of all the assets within a root
directory.
It's color code is as follows:
| Color | Meaning |
| ----- | --------------------------- |
| Cyan | Non-asset file or directory |
| Green | Valid asset |
| Red | Invalid asset |
## Config
The config tab is used for uploading and writing Hidebound's configuration file.
## API
The API tab is really a link to Hidebound's REST API documentation.
## Docs
The API tab is really a link to Hidebound's github documentation.
## Errors
Hidebound is oriented towards developers and technically proficient users. It
displays errors in their entirety within the application.
# Configuration
Hidebound is configured via a configuration file or environment variables.
Hidebound configs consist of four main sections:
## Base
* ingress_directory - the directory hidebound parses for assets that comprise its database
* staging_directory - the staging directory valid assets are created in
* specification_files - a list of python specification files
* include_regex - filepaths in the root that match this are included in the database
* exclude_regex - filepaths in the root that match this are excluded from the database
* write_mode - whether to copy or move files from root to staging
* redact_regex - regular expression which matches config keys whose valuse are to be redacted
* redact_hash - whether to redact config values with "REDACTED" or a hash of the value
* workflow - order list of steps to be followed in workflow
## Dask
Default configuration of Dask distributed framework.
* cluster_type - dask cluster type
* num_partitions - number of partions for each dataframe
* local_num_workers - number of workers on local cluster
* local_threads_per_worker - number of threads per worker on local cluster
* local_multiprocessing - use multiprocessing for local cluster
* gateway_address - gateway server address
* gateway_proxy_address - **scheduler** proxy server address
* gateway_public_address - gateway server address, as accessible from a web browser
* gateway_auth_type - authentication type
* gateway_api_token - api token or password
* gateway_api_user - api user
* gateway_cluster_options - list of dask gateway cluster options
* gateway_shutdown_on_close - whether to shudown cluster upon close
* gateway_timeout - gateway client timeout
## Exporters
Which exporters to us in the workflow.
Options include:
* s3
* disk
* girder
## Webhooks
Webhooks to call after the export phase has completed.
---
## Environment Variables
If `HIDEBOUND_CONFIG_FILEPATH` is set, Hidebound will ignore all other
environment variables and read the given filepath in as a yaml or json config
file.
| Variable | Format | Portion |
| ---------------------------------------- | ------ | -------------------------------------------------------- |
| HIDEBOUND_CONFIG_FILEPATH | str | Entire Hidebound config file |
| HIDEBOUND_INGRESS_DIRECTORY | str | ingress_directory parameter of config |
| HIDEBOUND_STAGING_DIRECTORY | str | staging_directory parameter of config |
| HIDEBOUND_INCLUDE_REGEX | str | include_regex parameter of config |
| HIDEBOUND_EXCLUDE_REGEX | str | exclude_regex parameter of config |
| HIDEBOUND_WRITE_MODE | str | write_mode parameter of config |
| HIDEBOUND_REDACT_REGEX | str | redact_regex parameter of config |
| HIDEBOUND_REDACT_HASH | str | redact_hash parameter of config |
| HIDEBOUND_WORKFLOW | yaml | workflow paramater of config |
| HIDEBOUND_SPECIFICATION_FILES | yaml | specification_files section of config |
| HIDEBOUND_DASK_CLUSTER_TYPE | str | dask cluster type |
| HIDEBOUND_DASK_NUM_PARTITIONS | int | number of partions for each dataframe |
| HIDEBOUND_DASK_LOCAL_NUM_WORKERS | int | number of workers on local cluster |
| HIDEBOUND_DASK_LOCAL_THREADS_PER_WORKER | int | number of threads per worker on local cluster |
| HIDEBOUND_DASK_LOCAL_MULTIPROCESSING | str | use multiprocessing for local cluster |
| HIDEBOUND_DASK_GATEWAY_ADDRESS | str | gateway server address |
| HIDEBOUND_DASK_GATEWAY_PROXY_ADDRESS | str | scheduler proxy server address |
| HIDEBOUND_DASK_GATEWAY_PUBLIC_ADDRESS | str | gateway server address, as accessible from a web browser |
| HIDEBOUND_DASK_GATEWAY_AUTH_TYPE | str | authentication type |
| HIDEBOUND_DASK_GATEWAY_API_TOKEN | str | api token or password |
| HIDEBOUND_DASK_GATEWAY_API_USER | str | api user |
| HIDEBOUND_DASK_GATEWAY_CLUSTER_OPTIONS | yaml | list of dask gateway cluster options |
| HIDEBOUND_DASK_GATEWAY_SHUTDOWN_ON_CLOSE | str | whether to shudown cluster upon close |
| HIDEBOUND_TIMEOUT | int | gateway client timeout |
| HIDEBOUND_EXPORTERS | yaml | exporters section of config |
| HIDEBOUND_WEBHOOKS | yaml | webhooks section of config |
| HIDEBOUND_TESTING | str | run in test mode |
---
## Config File
Here is a full example config with comments:
```yaml
ingress_directory: /mnt/storage/projects # where hb looks for assets
staging_directory: /mnt/storage/hidebound # hb staging directory
include_regex: "" # include files that match
exclude_regex: "\\.DS_Store" # exclude files that match
write_mode: copy # copy files from root to staging
# options: copy, move
redact_regex: "(_key|_id|_token|url)$" # regex matched config keys to redact
redact_hash: true # hash redacted values
workflow: # workflow steps
- delete # clear staging directory
- update # create database from ingress files
- create # stage valid assets
- export # export assets in staging
specification_files: # list of spec files
- /mnt/storage/specs/image_specs.py
- /mnt/storage/specs/video_specs.py
dask:
cluster_type: local # Dask cluster type
# options: local, gateway
num_partitions: 16 # number of partions for each dataframe
local_num_workers: 16 # number of workers on local cluster
local_threads_per_worker: 1 # number of threads per worker on local cluster
local_multiprocessing: true # use multiprocessing for local cluster
gateway_address: http://proxy-public/services/dask-gateway # gateway server address
gateway_proxy_address: gateway://traefik-daskhub-dask-gateway.core:80 # scheduler proxy server address
gateway_public_address: https://dask-gateway/services/dask-gateway/ # gateway server address, as accessible from a web browser
gateway_auth_type: jupyterhub # authentication type
gateway_api_token: token123 # api token or password
gateway_api_user: admin # api user
gateway_cluster_options: # list of dask gateway options
- field: image # option field
label: image # option label
option_type: select # options: bool, float, int, mapping, select, string
default: "some-image:latest" # option default value
options: # list of choices if option_type is select
- "some-image:latest" # choice 1
- "some-image:0.1.2" # choice 2
gateway_min_workers: 1 # min dask gateway workers
gateway_max_workers: 8 # max dask gateway workers
gateway_shutdown_on_close: true # whether to shudown cluster upon close
gateway_timeout: 30 # gateway client timeout
exporters: # dict of exporter configs
- name: disk # export to disk
target_directory: /mnt/storage/archive # target location
metadata_types: # options: asset, file, asset-chunk, file-chunk
- asset # only asset and file metadata
- file
dask: # dask settings override
num_workers: 8
local_threads_per_worker: 2
- name: s3 # export to s3
access_key: ABCDEFGHIJKLMNOPQRST # aws access key
secret_key: abcdefghijklmnopqrstuvwxyz1234567890abcd # aws secret key
bucket: prod-data # s3 bucket
region: us-west-2 # bucket region
metadata_types: # options: asset, file, asset-chunk, file-chunk
- asset # drop file metadata
- asset-chunk
- file-chunk
dask: # dask settings override
cluster_type: gateway
num_workers: 64
- name: girder # export to girder
api_key: eyS0nj9qPC5E7yK5l7nhGVPqDOBKPdA3EC60Rs9h # girder api key
root_id: 5ed735c8d8dd6242642406e5 # root resource id
root_type: collection # root resource type
host: http://prod.girder.com # girder server url
port: 8180 # girder server port
metadata_types: # options: asset, file
- asset # only asset metadata
dask: # dask settings override
num_workers: 10
dask: # dask settings override
num_workers: 10
webhooks: # call these after export
- url: https://hooks.slack.com/services/ABCDEFGHI/JKLMNO # slack URL
method: post # post this to slack
timeout: 60 # timeout after 60 seconds
# params: {} # params to post (NA here)
# json: {} # json to post (NA here)
data: # data to post
channel: "#hidebound" # slack data
text: export complete # slack data
username: hidebound # slack data
headers: # request headers
Content-type: application/json
```
# Specification
Asset specifications are defined in python using the base classes found in
specification_base.py. The base classes are defined using the schematics
framework. Hidebound generates a single JSON blob of metadata for each file of
an asset, and then combines blob into a single blob with a list values per key.
Thus every class member defined with schematics is encapsulated with ListType.
## Example asset
Suppose we have an image sequence asset that we wish to define a specificqtion
for. Our image sequences consist of a directory containing 1 or 3 channel png
with frame numbers in the filename.
```shell
projects
└── cat001
└── raw001
└── p-cat001_s-raw001_d-calico-jumping_v001
├── p-cat001_s-raw001_d-calico-jumping_v001_f0001.png
├── p-cat001_s-raw001_d-calico-jumping_v001_f0002.png
└── p-cat001_s-raw001_d-calico-jumping_v001_f0003.png
```
## Example specification
We would write the following specification for such an asset.
```python
from schematics.types import IntType, ListType, StringType
import hidebound.core.validators as vd # validates traits
import hidebound.core.traits as tr # gets properties of files and file names
from hidebound.core.specification_base import SequenceSpecificationBase
class Raw001(SequenceSpecificationBase):
asset_name_fields = [ # naming convention for asset directory
'project', 'specification', 'descriptor', 'version'
]
filename_fields = [ # naming convention for asset files
'project', 'specification', 'descriptor', 'version', 'frame',
'extension'
]
height = ListType(IntType(), required=True) # heights of png images
width = ListType(IntType(), required=True) # widths of png images
frame = ListType(
IntType(),
required=True,
validators=[vd.is_frame] # validates that frame is between 0 and 9999
)
channels = ListType(
IntType(),
required=True,
validators=[lambda x: vd.is_in(x, [1, 3])] # validates that png is 1 or 3 channel
)
extension = ListType(
StringType(),
required=True,
validators=[
vd.is_extension,
lambda x: vd.is_eq(x, 'png') # validates that image is png
]
)
file_traits = dict(
width=tr.get_image_width, # retrieves image width from file
height=tr.get_image_height, # retrieves image height from file
channels=tr.get_num_image_channels, # retrieves image channel number from file
)
```
# Production CLI
Hidebound comes with a command line interface defined in command.py.
Its usage pattern is: `hidebound COMMAND [FLAGS] [-h --help]`
## Commands
| Command | Description |
| --------------- | ------------------------------------------------------------------------- |
| bash-completion | Prints BASH completion code to be written to a _hidebound completion file |
| config | Prints hidebound config |
| serve | Runs a hidebound server |
| zsh-completion | Prints ZSH completion code to be written to a _hidebound completion file |
## Flags
| Command | Flag | Description | Default |
| ------- | --------- | ------------------------ | ------- |
| serve | --port | Server port | 8080 |
| serve | --timeout | Gunicorn timeout | 0 |
| serve | --testing | Testing mode | False |
| serve | --debug | Debug mode (no gunicorn) | False |
| all | --help | Show help message | <p></p> |
---
# Quickstart Guide
This repository contains a suite commands for the whole development process.
This includes everything from testing, to documentation generation and
publishing pip packages.
These commands can be accessed through:
- The VSCode task runner
- The VSCode task runner side bar
- A terminal running on the host OS
- A terminal within this repositories docker container
Running the `zsh-complete` command will enable tab completions of the CLI.
See the zsh setup section for more information.
## Command Groups
Development commands are grouped by one of 10 prefixes:
| Command | Description |
| ---------- | ---------------------------------------------------------------------------------- |
| build | Commands for building packages for testing and pip publishing |
| docker | Common docker commands such as build, start and stop |
| docs | Commands for generating documentation and code metrics |
| library | Commands for managing python package dependencies |
| session | Commands for starting interactive sessions such as jupyter lab and python |
| state | Command to display the current state of the repo and container |
| test | Commands for running tests, linter and type annotations |
| version | Commands for bumping project versions |
| quickstart | Display this quickstart guide |
| zsh | Commands for running a zsh session in the container and generating zsh completions |
## Common Commands
Here are some frequently used commands to get you started:
| Command | Description |
| ----------------- | --------------------------------------------------------- |
| docker-restart | Restart container |
| docker-start | Start container |
| docker-stop | Stop container |
| docs-full | Generate documentation, coverage report, diagram and code |
| library-add | Add a given package to a given dependency group |
| library-graph-dev | Graph dependencies in dev environment |
| library-remove | Remove a given package from a given dependency group |
| library-search | Search for pip packages |
| library-update | Update dev dependencies |
| session-lab | Run jupyter lab server |
| state | State of |
| test-dev | Run all tests |
| test-lint | Run linting and type checking |
| zsh | Run ZSH session inside container |
| zsh-complete | Generate ZSH completion script |
---
# Development CLI
bin/hidebound is a command line interface (defined in cli.py) that
works with any version of python 2.7 and above, as it has no dependencies.
Commands generally do not expect any arguments or flags.
Its usage pattern is: `bin/hidebound COMMAND [-a --args]=ARGS [-h --help] [--dryrun]`
## Commands
The following is a complete list of all available development commands:
| Command | Description |
| ----------------------- | ------------------------------------------------------------------- |
| build-package | Build production version of repo for publishing |
| build-prod | Publish pip package of repo to PyPi |
| build-publish | Run production tests first then publish pip package of repo to PyPi |
| build-test | Build test version of repo for prod testing |
| docker-build | Build Docker image |
| docker-build-from-cache | Build Docker image from cached image |
| docker-build-prod | Build production image |
| docker-container | Display the Docker container id |
| docker-destroy | Shutdown container and destroy its image |
| docker-destroy-prod | Shutdown production container and destroy its image |
| docker-image | Display the Docker image id |
| docker-prod | Start production container |
| docker-pull-dev | Pull development image from Docker registry |
| docker-pull-prod | Pull production image from Docker registry |
| docker-push-dev | Push development image to Docker registry |
| docker-push-dev-latest | Push development image to Docker registry with dev-latest tag |
| docker-push-prod | Push production image to Docker registry |
| docker-push-prod-latest | Push production image to Docker registry with prod-latest tag |
| docker-remove | Remove Docker image |
| docker-restart | Restart Docker container |
| docker-start | Start Docker container |
| docker-stop | Stop Docker container |
| docs | Generate sphinx documentation |
| docs-architecture | Generate architecture.svg diagram from all import statements |
| docs-full | Generate documentation, coverage report, diagram and code |
| docs-metrics | Generate code metrics report, plots and tables |
| library-add | Add a given package to a given dependency group |
| library-graph-dev | Graph dependencies in dev environment |
| library-graph-prod | Graph dependencies in prod environment |
| library-install-dev | Install all dependencies into dev environment |
| library-install-prod | Install all dependencies into prod environment |
| library-list-dev | List packages in dev environment |
| library-list-prod | List packages in prod environment |
| library-lock-dev | Resolve dev.lock file |
| library-lock-prod | Resolve prod.lock file |
| library-remove | Remove a given package from a given dependency group |
| library-search | Search for pip packages |
| library-sync-dev | Sync dev environment with packages listed in dev.lock |
| library-sync-prod | Sync prod environment with packages listed in prod.lock |
| library-update | Update dev dependencies |
| library-update-pdm | Update PDM |
| quickstart | Display quickstart guide |
| session-lab | Run jupyter lab server |
| session-python | Run python session with dev dependencies |
| session-server | Runn application server inside Docker container |
| state | State of repository and Docker container |
| test-coverage | Generate test coverage report |
| test-dev | Run all tests |
| test-fast | Test all code excepts tests marked with SKIP_SLOWS_TESTS decorator |
| test-lint | Run linting and type checking |
| test-prod | Run tests across all support python versions |
| version | Full resolution of repo: dependencies, linting, tests, docs, etc |
| version-bump-major | Bump pyproject major version |
| version-bump-minor | Bump pyproject minor version |
| version-bump-patch | Bump pyproject patch version |
| version-commit | Tag with version and commit changes to master |
| zsh | Run ZSH session inside Docker container |
| zsh-complete | Generate oh-my-zsh completions |
| zsh-root | Run ZSH session as root inside Docker container |
## Flags
| Short | Long | Description |
| ----- | --------- | ---------------------------------------------------- |
| -a | --args | Additional arguments, this can generally be ignored |
| -h | --help | Prints command help message to stdout |
| | --dryrun | Prints command that would otherwise be run to stdout |
Raw data
{
"_id": null,
"home_page": null,
"name": "hidebound",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "ephemeral, database, asset, assetstore, datastore, vfx, mlops, ml, machine-learning, data",
"author": null,
"author_email": "Alex Braun <alexander.g.braun@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/75/82/89c6829b88291cdb7f60200ff8763d86ab7e5dbf59b9a2bd10adae3816d8/hidebound-0.34.2.tar.gz",
"platform": null,
"description": "<p>\n <a href=\"https://www.linkedin.com/in/alexandergbraun\" rel=\"nofollow noreferrer\">\n <img src=\"https://www.gomezaparicio.com/wp-content/uploads/2012/03/linkedin-logo-1-150x150.png\"\n alt=\"linkedin\" width=\"30px\" height=\"30px\"\n >\n </a>\n <a href=\"https://github.com/theNewFlesh\" rel=\"nofollow noreferrer\">\n <img src=\"https://tadeuzagallo.com/GithubPulse/assets/img/app-icon-github.png\"\n alt=\"github\" width=\"30px\" height=\"30px\"\n >\n </a>\n <a href=\"https://pypi.org/user/the-new-flesh\" rel=\"nofollow noreferrer\">\n <img src=\"https://cdn.iconscout.com/icon/free/png-256/python-2-226051.png\"\n alt=\"pypi\" width=\"30px\" height=\"30px\"\n >\n </a>\n <a href=\"http://vimeo.com/user3965452\" rel=\"nofollow noreferrer\">\n <img src=\"https://cdn.iconscout.com/icon/free/png-512/movie-52-151107.png?f=avif&w=512\"\n alt=\"vimeo\" width=\"30px\" height=\"30px\"\n >\n </a>\n <a href=\"http://www.alexgbraun.com\" rel=\"nofollow noreferrer\">\n <img src=\"https://i.ibb.co/fvyMkpM/logo.png\"\n alt=\"alexgbraun\" width=\"30px\" height=\"30px\"\n >\n </a>\n</p>\n\n<!-- <img id=\"logo\" src=\"resources/logo.png\" style=\"max-width: 717px\"> -->\n\n[](https://github.com/theNewFlesh/hidebound/blob/master/LICENSE)\n[](https://github.com/theNewFlesh/hidebound/blob/master/docker/config/pyproject.toml)\n[](https://pypi.org/project/hidebound/)\n[](https://pepy.tech/project/hidebound)\n\n# Overview\nA MLOps framework for generating ML assets and metadata.\n\nHidebound is an ephemeral database and asset framework used for generating,\nvalidating and exporting assets to various data stores. Hidebound enables\ndevelopers to ingest arbitrary sets of files and output them as content and\ngenerated metadata, which has been validated according to specifications they\ndefine.\n\nAssets are placed into an ingress directory, typically reserved for Hidebound\nprojects, and then processed by Hidebound. Hidebound extracts metadata from the\nfiles and directories that make each asset according to their name, location and\nfile properties. This data comprises the entirety of Hidebound's database at any\none time.\n\nSee [documentation](https://thenewflesh.github.io/hidebound/) for details.\n\n# Installation for Developers\n\n### Docker\n1. Install [docker-desktop](https://docs.docker.com/desktop/)\n2. Ensure docker-desktop has at least 4 GB of memory allocated to it.\n3. `git clone git@github.com:theNewFlesh/hidebound.git`\n4. `cd hidebound`\n5. `chmod +x bin/hidebound`\n6. `bin/hidebound docker-start`\n - If building on a M1 Mac run `export DOCKER_DEFAULT_PLATFORM=linux/amd64` first.\n\nThe service should take a few minutes to start up.\n\nRun `bin/hidebound --help` for more help on the command line tool.\n\n### ZSH Setup\n1. `bin/hidebound` must be run from this repository's top level directory.\n2. Therefore, if using zsh, it is recommended that you paste the following line\n in your ~/.zshrc file:\n - `alias hidebound=\"cd [parent dir]/hidebound; bin/hidebound\"`\n - Replace `[parent dir]` with the parent directory of this repository\n3. Consider adding the following line to your ~/.zshrc if you are using a M1 Mac:\n - `export DOCKER_DEFAULT_PLATFORM=linux/amd64`\n4. Running the `zsh-complete` command will enable tab completions of the cli\n commands, in the next shell session.\n\n For example:\n - `hidebound [tab]` will show you all the cli options, which you can press\n tab to cycle through\n - `hidebound docker-[tab]` will show you only the cli options that begin with\n \"docker-\"\n\n# Installation for Production\n\n### Python\n`pip install hidebound`\n\n### Docker\n1. Install [docker-desktop](https://docs.docker.com/desktop/)\n2. `docker pull thenewflesh/hidebound:[version]`\n\n---\n\n# Dataflow\n\n\nData begins as files on disk. Hidebound creates a JSON-compatible dict from\ntheir name traits and file traits and then constructs an internal database table\nfrom them, one dict per row. All the rows are then aggregated by asset, and\nconverted into JSON blobs. Those blobs are then validated according to their\nrespective specifications. Files from valid assets are then copied or moved into\nHidebound's content directory, according to their same directory structure and\nnaming. Metadata is written to JSON files inside Hidebound's metadata directory.\nEach file's metadata is written as a JSON file in /hidebound/metadata/file, and\neach asset's metadata (the aggregate of its file metadata) is written to\n/hidebound/metadata/asset. From their exporters, can export the valid\nasset data and its accompanying metadata to various locations, like an AWS S3\nbucket.\n\n# Workflow\nThe acronynm to remember for workflows is **CRUDES**: create, read, update,\ndelete, export and search. Those operations constitue the main functionality\nthat Hidebound supports.\n\n## *Create Asset*\nFor example, an asset could be an image sequence, such as a directory full of\nPNG files, all of which have a frame number, have 3 (RGB) channels, and are 1024\npixels wide by 1024 pixels tall. Let's call the specification for this type of\nasset \"spec001\". We create an image sequence of a cat running, and we move it\ninto the Hidebound projects directory.\n\n## *Update*\n\n\nWe call the update function via Hidebound's web app. Hidebound creates\na new database based upon the recursive listing of all the files within said\ndirectory. This database is displayed to us as a table, with one file per row.\nIf we choose to group by asset in the app, the table will display one asset per\nrow. Hidebound extracts metadata from each filename (not any directory name) as\nwell as from the file itself. That metadata is called file_traits. Using only\ninformation derived from filename and file traits, Hidebound determines which\nfiles are grouped together as a single asset and the specification of that\nasset. Asset traits are then derived from this set of files (one or more).\nFinally, Hidebound validates each asset according to its determined\nspecification. All of this data is displayed as a table within the web app.\nImportantly, all of the errors in filenames, file traits and asset traits are\nincluded.\n\n## *Review Graph*\n\nIf we click on the graph tab, we are greeted by a hierarchical graph of all our\nassets in our project directory. Our asset is red, meaning it's invalid. Valid\nasset's are green, and all other files and directories, including parent\ndirectories, are cyan.\n\n## *Diagnose and Repair*\nWe flip back to the data tab. Using table within it, we search (via SQL) for our\nasset within Hidebound's freshly created database. We see an error in one of the\nfilenames, conveniently displayed in red text. The descriptor in one orf our\nfilenames has capital letters in it. This violates Hidebound's naming\nconvention, and so we get an error. We go and rename the file appropriately and\ncall update again. Our asset is now valid. The filenames are correct and we can\nsee in the height and width columns, that it's 1024 by 1024 and the channels\ncolumn says it has three.\n\n## *Create*\nNext we click the create button. For each valid asset, Hidebound generates file\nand asset metadata as JSON files within the hidebound/metadata directory.\nHidebound also copies or moves, depending on the config write mode, valid files\nand directories into the hidebound/content directory. Hidebound/content and\nhidebound/metadata are both staging directories used for generating a valid\nephemeral database. We now have a hidebound directory that looks like this\n(unmentioned assets are collapsed behind the ellipses):\n```shell\n/tmp/hidebound\n\u251c\u2500\u2500 hidebound_config.yaml\n\u2502\n\u251c\u2500\u2500 specifications\n\u2502 \u2514\u2500\u2500 specifications.py\n\u2502\n\u251c\u2500\u2500 data\n\u2502 ...\n\u2502 \u2514\u2500\u2500 p-cat001\n\u2502 \u2514\u2500\u2500 spec001\n\u2502 \u2514\u2500\u2500 p-cat001_s-spec001_d-running-cat_v001\n\u2502 \u251c\u2500\u2500 p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0001.png\n\u2502 \u251c\u2500\u2500 p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0002.png\n\u2502 \u2514\u2500\u2500 p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png\n\u2502\n\u251c\u2500\u2500 metadata\n \u251c\u2500\u2500 asset\n \u2502 ...\n \u2502 \u2514\u2500\u2500 a9f3727c-cb9b-4eb1-bc84-a6bc3b756cc5.json\n \u2502\n \u2514\u2500\u2500 file\n ...\n \u251c\u2500\u2500 279873a2-bfd0-4757-abf2-7dc4f771f992.json\n \u251c\u2500\u2500 e50160ae-8678-40b3-b766-ee8311b1f0c9.json\n \u2514\u2500\u2500 ea95bd79-cb8f-4262-8489-efe734c5f65c.json\n```\n\n## *Export*\nThe hidebound directories contain only valid assets. Thus, we are now free to\nexport this data to various data stores, such as AWS S3, MongoDB, and Girder.\nExporters are are defined within the exporters subpackage. They expect a\npopulated hidebound directory and use the files and metadata therein to export\nhidebound data. Exporter configurations are stored in the hidebound config,\nunder the \"exporters\" key. Currently supported exporters include, disk, s3 and\ngirder. Below we can see the results of an export to Girder in the Girder web\napp.\n\n\n\n## *Delete*\nOnce this export process is complete, we may click the delete button. Hidebound\ndeletes the hidebound/content and hidebound/metdata directories and all their\ncontents. If write_mode in the Hidebound configuration is set to \"copy\", then\nthis step will merely delete data created by Hidebound. If it is set to \"move\",\nthen Hidebound will presumably delete, the only existing copy of out asset data\non the host machine. The delete stage in combination with the removal of assets\nfrom the ingress directory is what makes Hidebound's database ephemeral.\n\n## *Workflow*\n`/api/workflow` is a API endpoint that initializes a database a with a given\nconfig, and then calls each method from a given list. For instance, if you send\nthis data to `/api/workflow`:\n\n```{config={...}, workflow=['update', 'create', 'export', 'delete']}```\n\nA database instance will be created with the given config, and then that\ninstance will call its update, create, export and delete methods, in that order.\n\n# Naming Convention\nHidebound is a highly opinionated framework that relies upon a strict but\ncomposable naming convention in order to extract metadata from filenames. All\nfiles and directories that are part of assets must conform to a naming\nconvention defined within that asset's specification.\n\nIn an over-simplified sense; sentences are constructions of words. Syntax\nconcerns how each word is formed, grammar concerns how to form words into a\nsentence, and semantics concerns what each word means. Similarly, filenames can\nbe thought of as crude sentences. They are made of several words (ie fields).\nThese words have distinct semantics (as determines by field indicators). Each\nword is constructed according to a syntax (ie indicator + token). All words are\njoined together by spaces (ie underscores) in a particular order as determined\nby grammar (as defined in each specification).\n\n## *Syntax*\n- Names consist of a series of fields, each separated by a single underscore\n \u201c_\u201d, also called a field separator.\n- Periods, \".\", are the exception to this, as it indicates file extension.\n- Legal characters include and only include:\n\n| Name | Characters | Use |\n| ---------------- | ---------- | ------------------------- |\n| Underscore | _ | only for field separation |\n| Period | . | only for file extensions |\n| Lowercase letter | a to z | everything |\n| Number | 0 to 9 | everything |\n| Hyphen | - | token separator |\n\nFields are comprised of two main parts:\n\n| Name | Use |\n| ---------------- | --------------------------------------------------- |\n| Field indicator | determines metadata key |\n| Field token | a set of 1+ characters that define the field's data |\n\n---\n## **Example Diagrams**\nIn our example filename:\n`p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png` the metadata will be:\n\n```json\n{\n \"project\": \"cat001\",\n \"specification\": \"spec001\",\n \"descriptor\": \"running-cat\",\n \"version\": 1,\n \"coordinate\": [0, 5],\n \"frame\": 3,\n \"extension\": \"png\",\n}\n```\n\nThe spec001 specification is derived from the second field of this filename:\n```shell\n field field\n indicator token\n | __|__\n | | |\np-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png\n |_______|\n |\n field\n```\n\n| Part | Value |\n| ---------------- | ------------------------ |\n| Field | s-spec001 |\n| Field indicator | s- |\n| Field token | spec001 |\n| Derived metadata | {specification: spec001} |\n\n## *Special Field Syntax*\n\n- Projects begin with 3 to 10 letters followed by 1 to 4 numbers\n- Specifications begin with 3 or 4 letters followed by 3 numbers\n- Descriptors begin with a letter or number and may also contain hyphens\n- Descriptors may not begin with the words master, final or last\n- Versions are triple-padded with zeros and must be greater than 0\n- Coordinates may contain up to 3 quadruple-padded numbers, separated by hyphens\n- Coordinates are always evaluated in XYZ order. For example: `c0001-0002-0003`\n produces `{x: 1, y: 2, z: 3}`.\n- Each element of a coordinate may be equal to or greater than zero\n- Frames are quadruple-padded and are greater than or equal to 0\n- Extensions may only contain upper and lower case letters a to z and numbers 0\n to 9\n\n## *Semantics*\nHidebound is highly opionated, especially with regards to its semantics. It\ncontains exactly seven field types, as indicated by their field indicators.\nThey are:\n\n| Field | Indicator |\n| ------------- | --------- |\n| project | p- |\n| specification | s- |\n| descriptor | d- |\n| version | v |\n| coordinate | c |\n| frame | f |\n| extension | . |\n\n## *Grammar*\nThe grammar is fairly simple:\n\n - Names are comprised of an ordered set of fields drawn from the seven above\n - All names must contain the specification field\n - All specification must define a field order\n - All fields of a name under that specification must occcur in its defined\n field order\n\nIts is highly encouraged that fields be defined in the following order:\n\n`project specification descriptor version coordinate frame extension`\n\nThe grammatical concept of field order here is one of rough encapsulation:\n\n- Projects contain assets\n- Assets are grouped by specification\n- A set of assets of the same content is grouped by a descriptor\n- That set of assets consists of multiple versions of the same content\n- A single asset may broken into chunks, identified by 1, 2 or 3 coordinates\n- Each chunk may consist of a series of files seperated by frame number\n- Each file has an extension\n\n## *Encouraged Lexical Conventions*\n- Specifications end with a triple padded number so that they may be explicitely\n versioned. You redefine an asset specification to something slightly\n different, by copying its specification class, adding one to its name and\n change the class attributes in some way. That way you always maintain\n backwards compatibility with legacy assets.\n- Descriptors are not a dumping ground for useless terms like wtf, junk, stuff,\n wip and test.\n- Descriptors should not specify information known at the asset specification\n level, such as the project name, the generic content of the asset (ie image,\n mask, png, etc).\n- Descriptors should not include information that can be known from the\n preceding tokens, such as version, frame or extension.\n- A descriptor should be applicable to every version of the asset it designates.\n- Use of hyphens in descriptors is encouraged.\n- When in doubt, hyphenate and put into the descriptor.\n\n---\n# Project Structure\nHidebound does not formally define a project structure. It merely stipulates\nthat assets must exist under some particular root directory. Each asset\nspecification does define a directory structure for the files that make up that\nasset. Assets are divided into 3 types: file, sequence and complex. File defines\nan asset that consists of a single file. Sequence is defined to be a single\ndirectory containing one or more files. Complex is for assets that consist of an\narbitrarily complex layout of directories and files.\n\nThe following project structure is recommended:\n\n```shell\nproject\n |-- specification\n |-- descriptor\n |-- asset # either a file or directory of files and directories\n |- file\n```\n\n## For Example\n```shell\n/tmp/projects\n\u2514\u2500\u2500 p-cat001\n \u251c\u2500\u2500 s-spec002\n \u2502 \u251c\u2500\u2500 d-calico-jumping\n \u2502 \u2502 \u2514\u2500\u2500 p-cat001_s-spec002_d-calico-jumping_v001\n \u2502 \u2502 \u251c\u2500\u2500 p-cat001_s-spec002_d-calico-jumping_v001_f0001.png\n \u2502 \u2502 \u251c\u2500\u2500 p-cat001_s-spec002_d-calico-jumping_v001_f0002.png\n \u2502 \u2502 \u2514\u2500\u2500 p-cat001_s-spec002_d-calico-jumping_v001_f0003.png\n \u2502 \u2502\n \u2502 \u2514\u2500\u2500 d-tabby-playing\n \u2502 \u251c\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v001\n \u2502 \u2502 \u251c\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v001_f0001.png\n \u2502 \u2502 \u251c\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v001_f0002.png\n \u2502 \u2502 \u2514\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v001_f0003.png\n \u2502 \u2502\n \u2502 \u2514\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v002\n \u2502 \u251c\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v002_f0001.png\n \u2502 \u251c\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v002_f0002.png\n \u2502 \u2514\u2500\u2500 p-cat001_s-spec002_d-tabby-playing_v002_f0003.png\n \u2502\n \u2514\u2500\u2500 spec001\n \u2514\u2500\u2500 p-cat001_s-spec001_d-running-cat_v001\n \u251c\u2500\u2500 p-cat001_s-spec001_d-Running-Cat_v001_c0000-0005_f0002.png\n \u251c\u2500\u2500 p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0001.png\n \u2514\u2500\u2500 p-cat001_s-spec001_d-running-cat_v001_c0000-0005_f0003.png\n```\n\n# Application\nThe Hidebound web application has five sections: data, graph, config, api and\ndocs.\n\n## Data\nThe data tab is the workhorse of the Hidebound app.\n\n\n\nIts functions are as follows:\n\n* Search - Search the updated database's data via SQL\n* Dropdown - Groups search results by file or asset\n* Init - Initialized the database with the current config\n* Update - Initializes and updates the database with the current config\n* Create - Copies or moves valid assets to hidebound/content directory and\n creates JSON files in hidebound/metadata directory\n* Delete - Deletes hidebound/content and hidebound/metadata directories\n\nPrior to calling update, the application will look like this:\n\n\n\n## Graph\nThe graph tab is used for visualizing the state of all the assets within a root\ndirectory.\n\n\n\nIt's color code is as follows:\n\n| Color | Meaning |\n| ----- | --------------------------- |\n| Cyan | Non-asset file or directory |\n| Green | Valid asset |\n| Red | Invalid asset |\n\n## Config\nThe config tab is used for uploading and writing Hidebound's configuration file.\n\n\n\n## API\nThe API tab is really a link to Hidebound's REST API documentation.\n\n\n\n## Docs\nThe API tab is really a link to Hidebound's github documentation.\n\n\n\n## Errors\nHidebound is oriented towards developers and technically proficient users. It\ndisplays errors in their entirety within the application.\n\n\n\n# Configuration\nHidebound is configured via a configuration file or environment variables.\n\nHidebound configs consist of four main sections:\n\n## Base\n* ingress_directory - the directory hidebound parses for assets that comprise its database\n* staging_directory - the staging directory valid assets are created in\n* specification_files - a list of python specification files\n* include_regex - filepaths in the root that match this are included in the database\n* exclude_regex - filepaths in the root that match this are excluded from the database\n* write_mode - whether to copy or move files from root to staging\n* redact_regex - regular expression which matches config keys whose valuse are to be redacted\n* redact_hash - whether to redact config values with \"REDACTED\" or a hash of the value\n* workflow - order list of steps to be followed in workflow\n\n## Dask\nDefault configuration of Dask distributed framework.\n\n* cluster_type - dask cluster type\n* num_partitions - number of partions for each dataframe\n* local_num_workers - number of workers on local cluster\n* local_threads_per_worker - number of threads per worker on local cluster\n* local_multiprocessing - use multiprocessing for local cluster\n* gateway_address - gateway server address\n* gateway_proxy_address - **scheduler** proxy server address\n* gateway_public_address - gateway server address, as accessible from a web browser\n* gateway_auth_type - authentication type\n* gateway_api_token - api token or password\n* gateway_api_user - api user\n* gateway_cluster_options - list of dask gateway cluster options\n* gateway_shutdown_on_close - whether to shudown cluster upon close\n* gateway_timeout - gateway client timeout\n\n## Exporters\nWhich exporters to us in the workflow.\nOptions include:\n\n* s3\n* disk\n* girder\n\n## Webhooks\nWebhooks to call after the export phase has completed.\n\n---\n\n## Environment Variables\nIf `HIDEBOUND_CONFIG_FILEPATH` is set, Hidebound will ignore all other\nenvironment variables and read the given filepath in as a yaml or json config\nfile.\n\n| Variable | Format | Portion |\n| ---------------------------------------- | ------ | -------------------------------------------------------- |\n| HIDEBOUND_CONFIG_FILEPATH | str | Entire Hidebound config file |\n| HIDEBOUND_INGRESS_DIRECTORY | str | ingress_directory parameter of config |\n| HIDEBOUND_STAGING_DIRECTORY | str | staging_directory parameter of config |\n| HIDEBOUND_INCLUDE_REGEX | str | include_regex parameter of config |\n| HIDEBOUND_EXCLUDE_REGEX | str | exclude_regex parameter of config |\n| HIDEBOUND_WRITE_MODE | str | write_mode parameter of config |\n| HIDEBOUND_REDACT_REGEX | str | redact_regex parameter of config |\n| HIDEBOUND_REDACT_HASH | str | redact_hash parameter of config |\n| HIDEBOUND_WORKFLOW | yaml | workflow paramater of config |\n| HIDEBOUND_SPECIFICATION_FILES | yaml | specification_files section of config |\n| HIDEBOUND_DASK_CLUSTER_TYPE | str | dask cluster type |\n| HIDEBOUND_DASK_NUM_PARTITIONS | int | number of partions for each dataframe |\n| HIDEBOUND_DASK_LOCAL_NUM_WORKERS | int | number of workers on local cluster |\n| HIDEBOUND_DASK_LOCAL_THREADS_PER_WORKER | int | number of threads per worker on local cluster |\n| HIDEBOUND_DASK_LOCAL_MULTIPROCESSING | str | use multiprocessing for local cluster |\n| HIDEBOUND_DASK_GATEWAY_ADDRESS | str | gateway server address |\n| HIDEBOUND_DASK_GATEWAY_PROXY_ADDRESS | str | scheduler proxy server address |\n| HIDEBOUND_DASK_GATEWAY_PUBLIC_ADDRESS | str | gateway server address, as accessible from a web browser |\n| HIDEBOUND_DASK_GATEWAY_AUTH_TYPE | str | authentication type |\n| HIDEBOUND_DASK_GATEWAY_API_TOKEN | str | api token or password |\n| HIDEBOUND_DASK_GATEWAY_API_USER | str | api user |\n| HIDEBOUND_DASK_GATEWAY_CLUSTER_OPTIONS | yaml | list of dask gateway cluster options |\n| HIDEBOUND_DASK_GATEWAY_SHUTDOWN_ON_CLOSE | str | whether to shudown cluster upon close |\n| HIDEBOUND_TIMEOUT | int | gateway client timeout |\n| HIDEBOUND_EXPORTERS | yaml | exporters section of config |\n| HIDEBOUND_WEBHOOKS | yaml | webhooks section of config |\n| HIDEBOUND_TESTING | str | run in test mode |\n\n---\n\n## Config File\nHere is a full example config with comments:\n```yaml\ningress_directory: /mnt/storage/projects # where hb looks for assets\nstaging_directory: /mnt/storage/hidebound # hb staging directory\ninclude_regex: \"\" # include files that match\nexclude_regex: \"\\\\.DS_Store\" # exclude files that match\nwrite_mode: copy # copy files from root to staging\n # options: copy, move\nredact_regex: \"(_key|_id|_token|url)$\" # regex matched config keys to redact\nredact_hash: true # hash redacted values\nworkflow: # workflow steps\n - delete # clear staging directory\n - update # create database from ingress files\n - create # stage valid assets\n - export # export assets in staging\nspecification_files: # list of spec files\n - /mnt/storage/specs/image_specs.py\n - /mnt/storage/specs/video_specs.py\ndask:\n cluster_type: local # Dask cluster type\n # options: local, gateway\n num_partitions: 16 # number of partions for each dataframe\n local_num_workers: 16 # number of workers on local cluster\n local_threads_per_worker: 1 # number of threads per worker on local cluster\n local_multiprocessing: true # use multiprocessing for local cluster\n gateway_address: http://proxy-public/services/dask-gateway # gateway server address\n gateway_proxy_address: gateway://traefik-daskhub-dask-gateway.core:80 # scheduler proxy server address\n gateway_public_address: https://dask-gateway/services/dask-gateway/ # gateway server address, as accessible from a web browser\n gateway_auth_type: jupyterhub # authentication type\n gateway_api_token: token123 # api token or password\n gateway_api_user: admin # api user\n gateway_cluster_options: # list of dask gateway options\n - field: image # option field\n label: image # option label\n option_type: select # options: bool, float, int, mapping, select, string\n default: \"some-image:latest\" # option default value\n options: # list of choices if option_type is select\n - \"some-image:latest\" # choice 1\n - \"some-image:0.1.2\" # choice 2\n gateway_min_workers: 1 # min dask gateway workers\n gateway_max_workers: 8 # max dask gateway workers\n gateway_shutdown_on_close: true # whether to shudown cluster upon close\n gateway_timeout: 30 # gateway client timeout\nexporters: # dict of exporter configs\n - name: disk # export to disk\n target_directory: /mnt/storage/archive # target location\n metadata_types: # options: asset, file, asset-chunk, file-chunk\n - asset # only asset and file metadata\n - file\n dask: # dask settings override\n num_workers: 8\n local_threads_per_worker: 2\n - name: s3 # export to s3\n access_key: ABCDEFGHIJKLMNOPQRST # aws access key\n secret_key: abcdefghijklmnopqrstuvwxyz1234567890abcd # aws secret key\n bucket: prod-data # s3 bucket\n region: us-west-2 # bucket region\n metadata_types: # options: asset, file, asset-chunk, file-chunk\n - asset # drop file metadata\n - asset-chunk\n - file-chunk\n dask: # dask settings override\n cluster_type: gateway\n num_workers: 64\n - name: girder # export to girder\n api_key: eyS0nj9qPC5E7yK5l7nhGVPqDOBKPdA3EC60Rs9h # girder api key\n root_id: 5ed735c8d8dd6242642406e5 # root resource id\n root_type: collection # root resource type\n host: http://prod.girder.com # girder server url\n port: 8180 # girder server port\n metadata_types: # options: asset, file\n - asset # only asset metadata\n dask: # dask settings override\n num_workers: 10\n dask: # dask settings override\n num_workers: 10\nwebhooks: # call these after export\n - url: https://hooks.slack.com/services/ABCDEFGHI/JKLMNO # slack URL\n method: post # post this to slack\n timeout: 60 # timeout after 60 seconds\n # params: {} # params to post (NA here)\n # json: {} # json to post (NA here)\n data: # data to post\n channel: \"#hidebound\" # slack data\n text: export complete # slack data\n username: hidebound # slack data\n headers: # request headers\n Content-type: application/json\n```\n\n# Specification\nAsset specifications are defined in python using the base classes found in\nspecification_base.py. The base classes are defined using the schematics\nframework. Hidebound generates a single JSON blob of metadata for each file of\nan asset, and then combines blob into a single blob with a list values per key.\nThus every class member defined with schematics is encapsulated with ListType.\n\n## Example asset\n\nSuppose we have an image sequence asset that we wish to define a specificqtion\nfor. Our image sequences consist of a directory containing 1 or 3 channel png\nwith frame numbers in the filename.\n\n```shell\nprojects\n \u2514\u2500\u2500 cat001\n \u2514\u2500\u2500 raw001\n \u2514\u2500\u2500 p-cat001_s-raw001_d-calico-jumping_v001\n \u251c\u2500\u2500 p-cat001_s-raw001_d-calico-jumping_v001_f0001.png\n \u251c\u2500\u2500 p-cat001_s-raw001_d-calico-jumping_v001_f0002.png\n \u2514\u2500\u2500 p-cat001_s-raw001_d-calico-jumping_v001_f0003.png\n```\n\n## Example specification\n\nWe would write the following specification for such an asset.\n\n```python\nfrom schematics.types import IntType, ListType, StringType\nimport hidebound.core.validators as vd # validates traits\nimport hidebound.core.traits as tr # gets properties of files and file names\nfrom hidebound.core.specification_base import SequenceSpecificationBase\n\nclass Raw001(SequenceSpecificationBase):\n asset_name_fields = [ # naming convention for asset directory\n 'project', 'specification', 'descriptor', 'version'\n ]\n filename_fields = [ # naming convention for asset files\n 'project', 'specification', 'descriptor', 'version', 'frame',\n 'extension'\n ]\n height = ListType(IntType(), required=True) # heights of png images\n width = ListType(IntType(), required=True) # widths of png images\n frame = ListType(\n IntType(),\n required=True,\n validators=[vd.is_frame] # validates that frame is between 0 and 9999\n )\n channels = ListType(\n IntType(),\n required=True,\n validators=[lambda x: vd.is_in(x, [1, 3])] # validates that png is 1 or 3 channel\n )\n extension = ListType(\n StringType(),\n required=True,\n validators=[\n vd.is_extension,\n lambda x: vd.is_eq(x, 'png') # validates that image is png\n ]\n )\n file_traits = dict(\n width=tr.get_image_width, # retrieves image width from file\n height=tr.get_image_height, # retrieves image height from file\n channels=tr.get_num_image_channels, # retrieves image channel number from file\n )\n```\n\n# Production CLI\nHidebound comes with a command line interface defined in command.py.\n\nIts usage pattern is: `hidebound COMMAND [FLAGS] [-h --help]`\n\n## Commands\n\n| Command | Description |\n| --------------- | ------------------------------------------------------------------------- |\n| bash-completion | Prints BASH completion code to be written to a _hidebound completion file |\n| config | Prints hidebound config |\n| serve | Runs a hidebound server |\n| zsh-completion | Prints ZSH completion code to be written to a _hidebound completion file |\n\n## Flags\n\n| Command | Flag | Description | Default |\n| ------- | --------- | ------------------------ | ------- |\n| serve | --port | Server port | 8080 |\n| serve | --timeout | Gunicorn timeout | 0 |\n| serve | --testing | Testing mode | False |\n| serve | --debug | Debug mode (no gunicorn) | False |\n| all | --help | Show help message | <p></p> |\n\n---\n\n# Quickstart Guide\nThis repository contains a suite commands for the whole development process.\nThis includes everything from testing, to documentation generation and\npublishing pip packages.\n\nThese commands can be accessed through:\n\n - The VSCode task runner\n - The VSCode task runner side bar\n - A terminal running on the host OS\n - A terminal within this repositories docker container\n\nRunning the `zsh-complete` command will enable tab completions of the CLI.\nSee the zsh setup section for more information.\n\n## Command Groups\n\nDevelopment commands are grouped by one of 10 prefixes:\n\n| Command | Description |\n| ---------- | ---------------------------------------------------------------------------------- |\n| build | Commands for building packages for testing and pip publishing |\n| docker | Common docker commands such as build, start and stop |\n| docs | Commands for generating documentation and code metrics |\n| library | Commands for managing python package dependencies |\n| session | Commands for starting interactive sessions such as jupyter lab and python |\n| state | Command to display the current state of the repo and container |\n| test | Commands for running tests, linter and type annotations |\n| version | Commands for bumping project versions |\n| quickstart | Display this quickstart guide |\n| zsh | Commands for running a zsh session in the container and generating zsh completions |\n\n## Common Commands\n\nHere are some frequently used commands to get you started:\n\n| Command | Description |\n| ----------------- | --------------------------------------------------------- |\n| docker-restart | Restart container |\n| docker-start | Start container |\n| docker-stop | Stop container |\n| docs-full | Generate documentation, coverage report, diagram and code |\n| library-add | Add a given package to a given dependency group |\n| library-graph-dev | Graph dependencies in dev environment |\n| library-remove | Remove a given package from a given dependency group |\n| library-search | Search for pip packages |\n| library-update | Update dev dependencies |\n| session-lab | Run jupyter lab server |\n| state | State of |\n| test-dev | Run all tests |\n| test-lint | Run linting and type checking |\n| zsh | Run ZSH session inside container |\n| zsh-complete | Generate ZSH completion script |\n\n---\n\n# Development CLI\nbin/hidebound is a command line interface (defined in cli.py) that\nworks with any version of python 2.7 and above, as it has no dependencies.\nCommands generally do not expect any arguments or flags.\n\nIts usage pattern is: `bin/hidebound COMMAND [-a --args]=ARGS [-h --help] [--dryrun]`\n\n## Commands\nThe following is a complete list of all available development commands:\n\n| Command | Description |\n| ----------------------- | ------------------------------------------------------------------- |\n| build-package | Build production version of repo for publishing |\n| build-prod | Publish pip package of repo to PyPi |\n| build-publish | Run production tests first then publish pip package of repo to PyPi |\n| build-test | Build test version of repo for prod testing |\n| docker-build | Build Docker image |\n| docker-build-from-cache | Build Docker image from cached image |\n| docker-build-prod | Build production image |\n| docker-container | Display the Docker container id |\n| docker-destroy | Shutdown container and destroy its image |\n| docker-destroy-prod | Shutdown production container and destroy its image |\n| docker-image | Display the Docker image id |\n| docker-prod | Start production container |\n| docker-pull-dev | Pull development image from Docker registry |\n| docker-pull-prod | Pull production image from Docker registry |\n| docker-push-dev | Push development image to Docker registry |\n| docker-push-dev-latest | Push development image to Docker registry with dev-latest tag |\n| docker-push-prod | Push production image to Docker registry |\n| docker-push-prod-latest | Push production image to Docker registry with prod-latest tag |\n| docker-remove | Remove Docker image |\n| docker-restart | Restart Docker container |\n| docker-start | Start Docker container |\n| docker-stop | Stop Docker container |\n| docs | Generate sphinx documentation |\n| docs-architecture | Generate architecture.svg diagram from all import statements |\n| docs-full | Generate documentation, coverage report, diagram and code |\n| docs-metrics | Generate code metrics report, plots and tables |\n| library-add | Add a given package to a given dependency group |\n| library-graph-dev | Graph dependencies in dev environment |\n| library-graph-prod | Graph dependencies in prod environment |\n| library-install-dev | Install all dependencies into dev environment |\n| library-install-prod | Install all dependencies into prod environment |\n| library-list-dev | List packages in dev environment |\n| library-list-prod | List packages in prod environment |\n| library-lock-dev | Resolve dev.lock file |\n| library-lock-prod | Resolve prod.lock file |\n| library-remove | Remove a given package from a given dependency group |\n| library-search | Search for pip packages |\n| library-sync-dev | Sync dev environment with packages listed in dev.lock |\n| library-sync-prod | Sync prod environment with packages listed in prod.lock |\n| library-update | Update dev dependencies |\n| library-update-pdm | Update PDM |\n| quickstart | Display quickstart guide |\n| session-lab | Run jupyter lab server |\n| session-python | Run python session with dev dependencies |\n| session-server | Runn application server inside Docker container |\n| state | State of repository and Docker container |\n| test-coverage | Generate test coverage report |\n| test-dev | Run all tests |\n| test-fast | Test all code excepts tests marked with SKIP_SLOWS_TESTS decorator |\n| test-lint | Run linting and type checking |\n| test-prod | Run tests across all support python versions |\n| version | Full resolution of repo: dependencies, linting, tests, docs, etc |\n| version-bump-major | Bump pyproject major version |\n| version-bump-minor | Bump pyproject minor version |\n| version-bump-patch | Bump pyproject patch version |\n| version-commit | Tag with version and commit changes to master |\n| zsh | Run ZSH session inside Docker container |\n| zsh-complete | Generate oh-my-zsh completions |\n| zsh-root | Run ZSH session as root inside Docker container |\n\n## Flags\n\n| Short | Long | Description |\n| ----- | --------- | ---------------------------------------------------- |\n| -a | --args | Additional arguments, this can generally be ignored |\n| -h | --help | Prints command help message to stdout |\n| | --dryrun | Prints command that would otherwise be run to stdout |\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A MLOps framework for generating ML assets and metadata.",
"version": "0.34.2",
"project_urls": {
"documentation": "https://thenewflesh.github.io/hidebound",
"repository": "https://github.com/theNewFlesh/hidebound"
},
"split_keywords": [
"ephemeral",
" database",
" asset",
" assetstore",
" datastore",
" vfx",
" mlops",
" ml",
" machine-learning",
" data"
],
"urls": [
{
"comment_text": "project 3 to 10 chars update",
"digests": {
"blake2b_256": "b9bde7b0643590b9a725a76047a58c580ccde6210e2df9ef8f1120f8e21ade5c",
"md5": "ddf8a21338a036c7f50d10eb9d98ddf0",
"sha256": "9566de5f302e4689e07d40f3038625b3ddb3bba0e7fe57072faa80c7b2ee1eb7"
},
"downloads": -1,
"filename": "hidebound-0.34.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ddf8a21338a036c7f50d10eb9d98ddf0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 72038,
"upload_time": "2024-07-06T00:30:59",
"upload_time_iso_8601": "2024-07-06T00:30:59.425771Z",
"url": "https://files.pythonhosted.org/packages/b9/bd/e7b0643590b9a725a76047a58c580ccde6210e2df9ef8f1120f8e21ade5c/hidebound-0.34.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "project 3 to 10 chars update",
"digests": {
"blake2b_256": "758289c6829b88291cdb7f60200ff8763d86ab7e5dbf59b9a2bd10adae3816d8",
"md5": "77230629316b74ecf9ed4104e2adf50f",
"sha256": "9db80e4209b84bcb2b4575053b2829e1375784b8871b82f2f3e39162ab99b14d"
},
"downloads": -1,
"filename": "hidebound-0.34.2.tar.gz",
"has_sig": false,
"md5_digest": "77230629316b74ecf9ed4104e2adf50f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 71669,
"upload_time": "2024-07-06T00:31:02",
"upload_time_iso_8601": "2024-07-06T00:31:02.132978Z",
"url": "https://files.pythonhosted.org/packages/75/82/89c6829b88291cdb7f60200ff8763d86ab7e5dbf59b9a2bd10adae3816d8/hidebound-0.34.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-06 00:31:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "theNewFlesh",
"github_project": "hidebound",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "hidebound"
}
JSON
