# Description
This module contains command-line executables aka CLIs that take in input atlas pipeline datasets and push them into Nexus along with a resource properties payload, including:
- push_volumetric to create a VolumetricDataLayer resource payload and push it along with the corresponding volumetric input dataset files into Nexus.
- push_meshes to create a Mesh resource payload and push it along with the corresponding brain .OBJ mesh folder input dataset files into Nexus.
- push_cellrecords to create a CellRecordSerie resource payload and push it along with the corresponding Sonata hdf5 file input dataset files into Nexus.
- push_regionsummary to create a RegionSummary resource payload and push it along with the corresponding brain region metadata json input dataset files into Nexus.
If the Resource already exists in Nexus then it will be updated instead. Eventually push/update their linked atlasRelease and ontology resources. Tag all these resources with the input tag or, if not provided, with a timestamp.
Each CLI can process multiple files/directories at once.
The input datasets must be one of the datasets listed in the input configuration file given as the argument of the --config option. These datasets has to correspond to the ones generated by the atlas pipeline (these datasets are referenced on the Pipeline Products page.) The configuration yaml file content structure should look like this configuration example :
HierarchyJson:
hierarchy: hierarchy.json
hierarchy_l23split: hierarchy_l23split.json
GeneratedDatasetPath:
VolumetricFile:
annotation_hybrid: annotation_v2v3_hybrid.nrrd
annotation_l23split: annotation_l23split.nrrd
cell_densities: cell_densities
neuron_densities: neuron_densities
MeshFile:
brain_region_meshes_hybrid: brain_region_meshes_v2v3_hybrid
brain_region_meshes_l23split: brain_region_meshes_l23split
CellRecordsFile:
cell_records_sonata: cell_records_sonata.h5
MetadataFile:
metadata_parcellations_ccfv3_l23split: metadata_parcellations_ccfv3_l23split.json
The Resource property payload includes, in addition to various information on the dataset pushed and its content, some provenance informations:
- The contributor: @id of the user and organisation associated with the input Nexus token.
- From which datasets it derivates.
- Informations on the run stocked in the linked Activity resource: the run duration, the software used ...
These four CLIs are hierarchically grouped in the cli initialize-pusher-cli. This CLI allows the Initialisation of the Forge python framework to communicate with Nexus. The Forge will enable to build and push into Nexus the metadata payload along with the input dataset.
This means that before calling one of the three CLIs, initialize-pusher-cli must first be called.
Note: the --verbosity argument allows you to print in the console the last resource payload from the list of resource payloads that has been constructed from input datasets that will be pushed into Nexus. If only one dataset has been given as input then its corresponding resource payload will be printed.
# Install
`pip install "bba-data-push"`
# Examples
## Arguments for initialize-pusher-cli
Run the dataset pusher CLI starting by the Initialisation of the Forge python framework to communicate with Nexus
### Inputs
--verbose, -v : Verbosity option. If equal True, the last resource payload from the list of resource payloads that has been constructed from input datasets will be printed. (Optional : boolean).
--forge-config-file : Path to the configuration file used to instantiate the Forge. (Optional, default = "https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/notebooks/use-cases/prod-forge-nexus.yml").
--nexus-env : Nexus environment to use, can be 'dev', staging', 'prod' or the URL of a custom environment. (Optional, default="prod").
--nexus-org : The Nexus organisation to push into. (Optional, default='bbp').
--nexus-org : The Nexus project to push into. (Optional, default='atlas').
--nexus-token-file : Path to the text file containing the Nexus token.
## Arguments for push_volumetric
Create a 'VolumetricDataLayer', an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and updated instead.
This script has been designed to function with volumetric files generated by the Atlas pipeline.
### Inputs
--dataset-path : [multiple paths] The files or directories of file to push on Nexus.
--config : Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset.
--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure.
--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created.
--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas.
--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources.
--resource-tag : [string] Optional tag value with which to tag the resources. (Optional).
--link-regions-path : [path] Optional json file containing link between regions and resources (@ ids of mask and mesh for each brain region) to be extracted by the CLI push-regionsummary. If the file already exists it will be annotated else it will be created. (Optional).
## Arguments for push_meshes
Create a 'Mesh' , an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and updated
instead.
This script has been designed to function with brain region meshes generated by the Atlas pipeline.
### Inputs
--dataset-path : [multiple paths] The files or directories of file to push on Nexus.
--config : [path] Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset.
--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure.
--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created.
--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas.
--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources.
--resource-tag : [string] Optional tag value with which to tag the resources. (Optional).
--link-regions-path : [path] Optional json file containing link between regions and resources (@ ids of mask and mesh for each brain region) to be extracted by the CLI push-regionsummary. If the file already exists it will be annotated else it will be created. (Optional).
## Arguments for push_regionsummary
Create a 'RegionSummary', an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and
updated instead.
This script has been designed to function with metadata json files generated by the Atlas pipeline.
### Inputs
--dataset-path : [multiple paths] The files or directories of file to push on Nexus.
--config : [path] Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset.
--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure.
--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created.
--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas.
--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources.
--resource-tag : [string] Optional tag value with which to tag the resources. (Optional).
--link-regions-path : [path] Optional json file containing link between regions and resources (@ ids of mask and mesh for each brain region) to be extracted by the CLI push-regionsummary. If the file already exists it will be annotated else it will be created. (Optional).
## Arguments for push_cellrecords
Create a 'CellRecordSeries', an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and updated instead.
This script has been designed to function with sonata h5 files storing 3D brain cell positions and orientations and generated by the Atlas pipeline.
### Inputs
--dataset-path : [multiple paths] The files or directories of file to push on Nexus.
--config : Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset.
--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure.
--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created.
--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas.
--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources.
--resource-tag : [string] Optional tag value with which to tag the resources. (Optional).
# Acknowledgements
The development of this software was supported by funding to the Blue Brain Project, a
research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss
government’s ETH Board of the Swiss Federal Institutes of Technology.
For license and authors, see LICENSE.txt and AUTHORS.txt respectively.
Copyright (c) 2020-2024 Blue Brain Project/EPFL
Raw data
{
"_id": null,
"home_page": "https://github.com/BlueBrain/bbp-data-push",
"name": "blue-brain-data-push",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Blue Brain Project, EPFL",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/ef/4b/d9bd79a226dae5875e9c0c0e452850d227f6fc2c1479b2ad16b4a5893ecb/blue_brain_data_push-5.0.0.tar.gz",
"platform": null,
"description": "# Description\n\nThis module contains command-line executables aka CLIs that take in input atlas pipeline datasets and push them into Nexus along with a resource properties payload, including:\n\n- push_volumetric to create a VolumetricDataLayer resource payload and push it along with the corresponding volumetric input dataset files into Nexus.\n\n- push_meshes to create a Mesh resource payload and push it along with the corresponding brain .OBJ mesh folder input dataset files into Nexus.\n\n- push_cellrecords to create a CellRecordSerie resource payload and push it along with the corresponding Sonata hdf5 file input dataset files into Nexus.\n\n- push_regionsummary to create a RegionSummary resource payload and push it along with the corresponding brain region metadata json input dataset files into Nexus.\n\nIf the Resource already exists in Nexus then it will be updated instead. Eventually push/update their linked atlasRelease and ontology resources. Tag all these resources with the input tag or, if not provided, with a timestamp.\nEach CLI can process multiple files/directories at once. \n\nThe input datasets must be one of the datasets listed in the input configuration file given as the argument of the --config option. These datasets has to correspond to the ones generated by the atlas pipeline (these datasets are referenced on the Pipeline Products page.) The configuration yaml file content structure should look like this configuration example :\n\n HierarchyJson:\n hierarchy: hierarchy.json\n hierarchy_l23split: hierarchy_l23split.json\n GeneratedDatasetPath:\n VolumetricFile:\n annotation_hybrid: annotation_v2v3_hybrid.nrrd\n annotation_l23split: annotation_l23split.nrrd\n cell_densities: cell_densities\n neuron_densities: neuron_densities\n MeshFile:\n brain_region_meshes_hybrid: brain_region_meshes_v2v3_hybrid\n brain_region_meshes_l23split: brain_region_meshes_l23split\n CellRecordsFile:\n cell_records_sonata: cell_records_sonata.h5\n \t MetadataFile:\n \t metadata_parcellations_ccfv3_l23split: metadata_parcellations_ccfv3_l23split.json\n\n\nThe Resource property payload includes, in addition to various information on the dataset pushed and its content, some provenance informations:\n\n- The contributor: @id of the user and organisation associated with the input Nexus token.\n- From which datasets it derivates.\n- Informations on the run stocked in the linked Activity resource: the run duration, the software used ...\n\n\nThese four CLIs are hierarchically grouped in the cli initialize-pusher-cli. This CLI allows the Initialisation of the Forge python framework to communicate with Nexus. The Forge will enable to build and push into Nexus the metadata payload along with the input dataset.\nThis means that before calling one of the three CLIs, initialize-pusher-cli must first be called. \n\nNote: the --verbosity argument allows you to print in the console the last resource payload from the list of resource payloads that has been constructed from input datasets that will be pushed into Nexus. If only one dataset has been given as input then its corresponding resource payload will be printed.\n\n\n# Install\n`pip install \"bba-data-push\"`\n\n\n# Examples\n\n## Arguments for initialize-pusher-cli\n\nRun the dataset pusher CLI starting by the Initialisation of the Forge python framework to communicate with Nexus\n\n### Inputs\n--verbose, -v : Verbosity option. If equal True, the last resource payload from the list of resource payloads that has been constructed from input datasets will be printed. (Optional : boolean). \n--forge-config-file : Path to the configuration file used to instantiate the Forge. (Optional, default = \"https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/notebooks/use-cases/prod-forge-nexus.yml\"). \n--nexus-env : Nexus environment to use, can be 'dev', staging', 'prod' or the URL of a custom environment. (Optional, default=\"prod\"). \n--nexus-org : The Nexus organisation to push into. (Optional, default='bbp'). \n--nexus-org : The Nexus project to push into. (Optional, default='atlas'). \n--nexus-token-file : Path to the text file containing the Nexus token. \n\n\n## Arguments for push_volumetric\n\nCreate a 'VolumetricDataLayer', an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and updated instead.\nThis script has been designed to function with volumetric files generated by the Atlas pipeline. \n\n### Inputs\n--dataset-path : [multiple paths] The files or directories of file to push on Nexus. \n--config : Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset. \n--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure. \n--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created. \n--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas. \n--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources. \n--resource-tag : [string] Optional tag value with which to tag the resources. (Optional). \n--link-regions-path : [path] Optional json file containing link between regions and resources (@ ids of mask and mesh for each brain region) to be extracted by the CLI push-regionsummary. If the file already exists it will be annotated else it will be created. (Optional). \n\n\n## Arguments for push_meshes\n\nCreate a 'Mesh' , an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and updated \ninstead. \nThis script has been designed to function with brain region meshes generated by the Atlas pipeline.\n\n### Inputs\n--dataset-path : [multiple paths] The files or directories of file to push on Nexus. \n--config : [path] Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset. \n--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure. \n--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created. \n--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas. \n--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources. \n--resource-tag : [string] Optional tag value with which to tag the resources. (Optional). \n--link-regions-path : [path] Optional json file containing link between regions and resources (@ ids of mask and mesh for each brain region) to be extracted by the CLI push-regionsummary. If the file already exists it will be annotated else it will be created. (Optional). \n\n\n## Arguments for push_regionsummary\n\nCreate a 'RegionSummary', an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and \nupdated instead. \nThis script has been designed to function with metadata json files generated by the Atlas pipeline.\n\n### Inputs\n--dataset-path : [multiple paths] The files or directories of file to push on Nexus. \n--config : [path] Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset. \n--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure. \n--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created. \n--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas. \n--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources. \n--resource-tag : [string] Optional tag value with which to tag the resources. (Optional). \n--link-regions-path : [path] Optional json file containing link between regions and resources (@ ids of mask and mesh for each brain region) to be extracted by the CLI push-regionsummary. If the file already exists it will be annotated else it will be created. (Optional). \n\n\n## Arguments for push_cellrecords\n\nCreate a 'CellRecordSeries', an 'atlasRelease' and an 'ontology' resource payload to push into Nexus. If the resources already exist in Nexus, they will be fetched and updated instead. \nThis script has been designed to function with sonata h5 files storing 3D brain cell positions and orientations and generated by the Atlas pipeline.\n\n### Inputs\n--dataset-path : [multiple paths] The files or directories of file to push on Nexus. \n--config : Path to the generated dataset configuration file. This is a yaml file containing the paths to the Atlas pipeline generated dataset. \n--hierarchy-path : [multiple paths] The path to the json hierarchy file containing an AIBS hierarchy structure. \n--hierarchy-jsonld-path : [path] Path to the AIBS hierarchy structure as a JSON-LD file. It is mandatory in case of the creation of a new Ontology resource as it will be attached to it when integrated in the knowledge graph. New Ontology resource is created at the same time a new atlasRelease resource need to be created. \n--atlasrelease-config-path : [path] Json file containing the atlasRelease @id as well as its ontology and parcellation volume @id. It needs to contains at least these informations for the atlasRelease Allen Mouse CCFV2 and CCFV3 stocked in the Nexus project bbp/atlas. \n--provenance-metadata-path : [path] Path to the Json file containing metadata for the derivation properties as well as the Activity and SoftwareAgent resources. \n--resource-tag : [string] Optional tag value with which to tag the resources. (Optional). \n\n\n\n# Acknowledgements\n\nThe development of this software was supported by funding to the Blue Brain Project, a \nresearch center of the \u00c9cole polytechnique f\u00e9d\u00e9rale de Lausanne (EPFL), from the Swiss \ngovernment\u2019s ETH Board of the Swiss Federal Institutes of Technology.\n\nFor license and authors, see LICENSE.txt and AUTHORS.txt respectively.\n\nCopyright (c) 2020-2024 Blue Brain Project/EPFL\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Package creating resource payloads from atlas datasets and push them along with the corresponding dataset files into Nexus.",
"version": "5.0.0",
"project_urls": {
"Homepage": "https://github.com/BlueBrain/bbp-data-push"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "23d1ca74a1db59fe5ffcaf4448024019470ab38c0f0ce84c5b1f3ff8ffd1419c",
"md5": "9712dbcfa5d06fb238694adbc05e673f",
"sha256": "e3387b0a495999ab9d2ba7aaa99ab27185d9a045deb43a3d05a92b4a14b27ad3"
},
"downloads": -1,
"filename": "blue_brain_data_push-5.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9712dbcfa5d06fb238694adbc05e673f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 35926236,
"upload_time": "2024-11-08T14:01:03",
"upload_time_iso_8601": "2024-11-08T14:01:03.152935Z",
"url": "https://files.pythonhosted.org/packages/23/d1/ca74a1db59fe5ffcaf4448024019470ab38c0f0ce84c5b1f3ff8ffd1419c/blue_brain_data_push-5.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ef4bd9bd79a226dae5875e9c0c0e452850d227f6fc2c1479b2ad16b4a5893ecb",
"md5": "7e4fb51c647afd6a0f643df4e0dff750",
"sha256": "9790261108b7011c4d22aebb4863fbc0f5a5a4606d286c821495c923c0bba61d"
},
"downloads": -1,
"filename": "blue_brain_data_push-5.0.0.tar.gz",
"has_sig": false,
"md5_digest": "7e4fb51c647afd6a0f643df4e0dff750",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 35703774,
"upload_time": "2024-11-08T14:01:06",
"upload_time_iso_8601": "2024-11-08T14:01:06.449797Z",
"url": "https://files.pythonhosted.org/packages/ef/4b/d9bd79a226dae5875e9c0c0e452850d227f6fc2c1479b2ad16b4a5893ecb/blue_brain_data_push-5.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-08 14:01:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "BlueBrain",
"github_project": "bbp-data-push",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "blue-brain-data-push"
}
JSON
