# MARISCO
<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
The [IAEA **M**arine **R**adioactivity **I**nformation **S**ystem
(MARIS)](https://maris.iaea.org) provides open access to radioactivity
measurements in marine environments. Developed by the [IAEA Marine
Environmental
Laboratories](https://www.iaea.org/about/organizational-structure/department-of-nuclear-sciences-and-applications/division-of-iaea-environment-laboratories)
in Monaco, MARIS offers data on seawater, biota, sediment, and suspended
matter.
This Python package includes command-line tools to convert MARIS
datasets into [`NetCDF`](https://www.unidata.ucar.edu/software/netcdf/)
or `.csv` formats, enhancing compatibility with various scientific and
data analysis software.
## Core Concept: Handlers
`marisco` is built around the concept of `handlers` - specialized
modules designed to convert MARIS datasets into NetCDF format. Each
handler is tailored to a specific data provider and implemented as a
dedicated Jupyter notebook.
### Literate Programming Approach
We’ve adopted a Literate Programming approach, which means:
1. **Documentation**: Each handler serves as comprehensive
documentation.
2. **Code Reference**: The notebooks contain the actual implementation
code.
3. **Communication Tool**: They facilitate discussions with data
providers about discrepancies or inconsistencies.
### Powered by nbdev
To achieve this, we leverage [nbdev](https://nbdev.fast.ai), a powerful
tool that allows us to:
1. Write code within Jupyter notebooks
2. Automatically export relevant parts as dedicated Python modules
This approach bridges the gap between documentation and implementation,
ensuring they remain in sync.
### See It in Action
For a concrete example of this approach, check out our [OSPAR dataset
handler
implementation](https://fr.anckalbi.net/marisco/handlers/ospar.html).
### List of currently available handlers
MARISCO includes a suite of specialized data handlers designed to:
- Convert provider-specific data formats into standardized MARIS NetCDF
files
- Ensure data quality and consistency across providers
- Facilitate integration with the MARIS marine radioactivity database
- Support automated data processing workflows
The following handlers are currently implemented:
| Handler | Description | Link to Data Source |
|----|----|----|
| [MARIS Legacy](https://fr.anckalbi.net/marisco/handlers/maris_legacy.html) | All legacy MARIS datasets from the MARIS Master Database | \- |
| [HELCOM](https://fr.anckalbi.net/marisco/handlers/helcom.html) | HELCOM marine environment protection datasets | [HELCOM](https://helcom.fi/about-us) |
| [OSPAR](https://fr.anckalbi.net/marisco/handlers/ospar.html) | OSPAR marine environment datasets | [ODIMS OSPAR](https://odims.ospar.org/en/) |
| [TEPCO](https://fr.anckalbi.net/marisco/handlers/tepco.html) | TEPCO Fukushima monitoring data | [TEPCO Monitoring](https://radioactivity.nsr.go.jp/ja/list/349/list-1.html) |
| [GEOTRACES](https://fr.anckalbi.net/marisco/handlers/geotraces.html) | BODC GEOTRACES oceanographic data | [GEOTRACES IDP2021](https://www.geotraces.org/geotraces-intermediate-data-product-2021/) |
## Install
Now, to install `marisco` simply run
``` console
pip install marisco
```
Once successfully installed, run the following command:
``` console
maris_init
```
This command:
1. creates a `.marisco/` directory containing various
configuration/configurable files ((below)) in your `/home`
directory;
2. creates a `configs.toml` file containing default but configurable
settings (default paths, …);
3. downloads several MARIS DB nomenclature/lookup table into
`.marisco/lut/` directory;
4. downloads `maris-template.nc`, the MARIS NetCDF4 template.
### Zotero API key
Upon conversion, `marisco` will automatically retrieve the bibliographic
metadata of each MARIS dataset from [Zotero](https://www.zotero.org/).
To do so, you need to define the following environment variable
`ZOTERO_API_KEY` containing the MARIS Zotero API key. Please contact the
MARIS team to get your API key.
## Getting started
### Command line utilities
All commands accept a `-h` argument to get access to its documentation.
#### `maris_init`
Download configuration file, NetCDF MARIS template and required lookup
tables (nomenclatures).
#### `maris_to_nc`
Convert `helcom`, `geotraces`, `tepco` or `ospar` marine radioactivity
datasets to MARIS NetCDF4 format.
usage: maris_to_nc [-h] [--src SRC] ds dest
positional arguments:
ds Name of the dataset to encode as NetCDF4
dest Output path for NetCDF file
options:
-h, --help show this help message and exit
--src SRC Optional input data path only required for the 'GEOTRACES' dataset
For instance: `maris_to_nc ospar 191-OSPAR-2024.nc`
#### `maris_db_to_nc`
The MARIS Master Database integrates two types of datasets:
- Historical datasets retrieved from published scientific papers
- Ongoing monitoring data from international programs like `HELCOM`,
`OSPAR`, `TEPCO`, and `GEOTRACES`
This command-line utility converts MARIS datasets from their legacy
format to NetCDF4, making them more accessible for modern data analysis
workflows. Users can either convert the entire database or specify
particular datasets by their reference IDs for selective conversion.
usage: maris_db_to_nc [-h] [--ref_ids REF_IDS] src dest
Convert MARIS legacy database to NetCDF4 format. If ref_ids is provided as comma-separated values, only encodes those subsets.
positional arguments:
src Path to MARIS database dump as `.txt` file
dest Output path for NetCDF file(s)
options:
-h, --help show this help message and exit
--ref_ids REF_IDS Optional comma-separated reference IDs (e.g., "123,456,789") (default: )
For instance:
- `maris_db_to_nc "~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt" ~/pro/tmp/output`
- or
`maris_db_to_nc "~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt" ~/pro/tmp/output --ref_ids="16,30"`
for a subset of the MARIS Master Database.
#### `maris_nc_to_csv`
This utility converts NetCDF files to CSV files that conform to the
MARIS Standard format, originally designed for OpenRefine workflows.
Although MARISCO has now superseded OpenRefine in the data preparation
pipeline, the MARIS master database continues to require CSV inputs in
this legacy format. This command-line utility, built with the MARISCO
library, handles the conversion process.
usage: maris_nc_to_csv [-h] src dest
Converts NetCDF files into CSV files that follow the MARIS Standard format.
positional arguments:
src Input path and filename for NetCDF file
dest Output path and filename (without extension) for CSV file
options:
-h, --help show this help message and exit
For instance:
`maris_nc_to_csv ~/pro/tmp/output/191-OSPAR-2024.nc ~/pro/tmp/output/191-OSPAR-2024`
> [!TIP]
>
> ### Note
>
> When specifying the destination path (e.g.,
> `~/pro/tmp/output/191-OSPAR-2024`), the utility automatically appends
> the MARIS sample type to the filename. For example:
>
> - `191-OSPAR-2024_BIOTA.csv` for biological samples
>
> While this specific example produces only a BIOTA file, the utility
> can generate multiple files (one per sample type) depending on the
> content of the source dataset. This reflects the NetCDF4 file
> structure, where each MARIS sample type is stored as a separate group
> within the file.
## Development
The MARIS NetCDF template is generated from `nbs/files/cdl/maris.cdl`
Common Data Language (CDL) file as defined by
[Unidata](https://docs.unidata.ucar.edu/). To generate the MARIS NetCDF
template `nbs/files/nc/maris-template.nc`, install the
[NetCDF-C](https://pjbartlein.github.io/REarthSysSci/install_netCDF.html)
utilities, once in `Marisco` home directory, run:
``` console
ncgen -4 -o nc/maris-template.nc cdl/maris.cdl
```
Raw data
{
"_id": null,
"home_page": "https://github.com/franckalbinet/marisco",
"name": "marisco",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "nbdev jupyter notebook python netcdf marine radioactivity data",
"author": "Franck Albinet, Niall Murphy",
"author_email": "franckalbinet@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/35/ba/c5c4c5c58da78ba1611e015248145c53ddb736f44792eace12269ef52d8e/marisco-1.0.0.tar.gz",
"platform": null,
"description": "# MARISCO\n\n\n<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->\n\nThe [IAEA **M**arine **R**adioactivity **I**nformation **S**ystem\n(MARIS)](https://maris.iaea.org) provides open access to radioactivity\nmeasurements in marine environments. Developed by the [IAEA Marine\nEnvironmental\nLaboratories](https://www.iaea.org/about/organizational-structure/department-of-nuclear-sciences-and-applications/division-of-iaea-environment-laboratories)\nin Monaco, MARIS offers data on seawater, biota, sediment, and suspended\nmatter.\n\nThis Python package includes command-line tools to convert MARIS\ndatasets into [`NetCDF`](https://www.unidata.ucar.edu/software/netcdf/)\nor `.csv` formats, enhancing compatibility with various scientific and\ndata analysis software.\n\n## Core Concept: Handlers\n\n`marisco` is built around the concept of `handlers` - specialized\nmodules designed to convert MARIS datasets into NetCDF format. Each\nhandler is tailored to a specific data provider and implemented as a\ndedicated Jupyter notebook.\n\n### Literate Programming Approach\n\nWe\u2019ve adopted a Literate Programming approach, which means:\n\n1. **Documentation**: Each handler serves as comprehensive\n documentation.\n2. **Code Reference**: The notebooks contain the actual implementation\n code.\n3. **Communication Tool**: They facilitate discussions with data\n providers about discrepancies or inconsistencies.\n\n### Powered by nbdev\n\nTo achieve this, we leverage [nbdev](https://nbdev.fast.ai), a powerful\ntool that allows us to:\n\n1. Write code within Jupyter notebooks\n2. Automatically export relevant parts as dedicated Python modules\n\nThis approach bridges the gap between documentation and implementation,\nensuring they remain in sync.\n\n### See It in Action\n\nFor a concrete example of this approach, check out our [OSPAR dataset\nhandler\nimplementation](https://fr.anckalbi.net/marisco/handlers/ospar.html).\n\n### List of currently available handlers\n\nMARISCO includes a suite of specialized data handlers designed to:\n\n- Convert provider-specific data formats into standardized MARIS NetCDF\n files\n- Ensure data quality and consistency across providers\n- Facilitate integration with the MARIS marine radioactivity database\n- Support automated data processing workflows\n\nThe following handlers are currently implemented:\n\n| Handler | Description | Link to Data Source |\n|----|----|----|\n| [MARIS Legacy](https://fr.anckalbi.net/marisco/handlers/maris_legacy.html) | All legacy MARIS datasets from the MARIS Master Database | \\- |\n| [HELCOM](https://fr.anckalbi.net/marisco/handlers/helcom.html) | HELCOM marine environment protection datasets | [HELCOM](https://helcom.fi/about-us) |\n| [OSPAR](https://fr.anckalbi.net/marisco/handlers/ospar.html) | OSPAR marine environment datasets | [ODIMS OSPAR](https://odims.ospar.org/en/) |\n| [TEPCO](https://fr.anckalbi.net/marisco/handlers/tepco.html) | TEPCO Fukushima monitoring data | [TEPCO Monitoring](https://radioactivity.nsr.go.jp/ja/list/349/list-1.html) |\n| [GEOTRACES](https://fr.anckalbi.net/marisco/handlers/geotraces.html) | BODC GEOTRACES oceanographic data | [GEOTRACES IDP2021](https://www.geotraces.org/geotraces-intermediate-data-product-2021/) |\n\n## Install\n\nNow, to install `marisco` simply run\n\n``` console\npip install marisco\n```\n\nOnce successfully installed, run the following command:\n\n``` console\nmaris_init\n```\n\nThis command:\n\n1. creates a `.marisco/` directory containing various\n configuration/configurable files ((below)) in your `/home`\n directory;\n2. creates a `configs.toml` file containing default but configurable\n settings (default paths, \u2026);\n3. downloads several MARIS DB nomenclature/lookup table into\n `.marisco/lut/` directory;\n4. downloads `maris-template.nc`, the MARIS NetCDF4 template.\n\n### Zotero API key\n\nUpon conversion, `marisco` will automatically retrieve the bibliographic\nmetadata of each MARIS dataset from [Zotero](https://www.zotero.org/).\nTo do so, you need to define the following environment variable\n`ZOTERO_API_KEY` containing the MARIS Zotero API key. Please contact the\nMARIS team to get your API key.\n\n## Getting started\n\n### Command line utilities\n\nAll commands accept a `-h` argument to get access to its documentation.\n\n#### `maris_init`\n\nDownload configuration file, NetCDF MARIS template and required lookup\ntables (nomenclatures).\n\n#### `maris_to_nc`\n\nConvert `helcom`, `geotraces`, `tepco` or `ospar` marine radioactivity\ndatasets to MARIS NetCDF4 format.\n\n usage: maris_to_nc [-h] [--src SRC] ds dest\n\n positional arguments:\n ds Name of the dataset to encode as NetCDF4\n dest Output path for NetCDF file\n\n options:\n -h, --help show this help message and exit\n --src SRC Optional input data path only required for the 'GEOTRACES' dataset\n\nFor instance: `maris_to_nc ospar 191-OSPAR-2024.nc`\n\n#### `maris_db_to_nc`\n\nThe MARIS Master Database integrates two types of datasets:\n\n- Historical datasets retrieved from published scientific papers\n- Ongoing monitoring data from international programs like `HELCOM`,\n `OSPAR`, `TEPCO`, and `GEOTRACES`\n\nThis command-line utility converts MARIS datasets from their legacy\nformat to NetCDF4, making them more accessible for modern data analysis\nworkflows. Users can either convert the entire database or specify\nparticular datasets by their reference IDs for selective conversion.\n\n usage: maris_db_to_nc [-h] [--ref_ids REF_IDS] src dest\n\n Convert MARIS legacy database to NetCDF4 format. If ref_ids is provided as comma-separated values, only encodes those subsets.\n\n positional arguments:\n src Path to MARIS database dump as `.txt` file\n dest Output path for NetCDF file(s)\n\n options:\n -h, --help show this help message and exit\n --ref_ids REF_IDS Optional comma-separated reference IDs (e.g., \"123,456,789\") (default: )\n\nFor instance:\n\n- `maris_db_to_nc \"~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt\" ~/pro/tmp/output` \n- or\n `maris_db_to_nc \"~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt\" ~/pro/tmp/output --ref_ids=\"16,30\"`\n for a subset of the MARIS Master Database.\n\n#### `maris_nc_to_csv`\n\nThis utility converts NetCDF files to CSV files that conform to the\nMARIS Standard format, originally designed for OpenRefine workflows.\n\nAlthough MARISCO has now superseded OpenRefine in the data preparation\npipeline, the MARIS master database continues to require CSV inputs in\nthis legacy format. This command-line utility, built with the MARISCO\nlibrary, handles the conversion process.\n\n usage: maris_nc_to_csv [-h] src dest\n\n Converts NetCDF files into CSV files that follow the MARIS Standard format.\n\n positional arguments:\n src Input path and filename for NetCDF file\n dest Output path and filename (without extension) for CSV file\n\n options:\n -h, --help show this help message and exit\n\nFor instance:\n`maris_nc_to_csv ~/pro/tmp/output/191-OSPAR-2024.nc ~/pro/tmp/output/191-OSPAR-2024`\n\n> [!TIP]\n>\n> ### Note\n>\n> When specifying the destination path (e.g.,\n> `~/pro/tmp/output/191-OSPAR-2024`), the utility automatically appends\n> the MARIS sample type to the filename. For example:\n>\n> - `191-OSPAR-2024_BIOTA.csv` for biological samples\n>\n> While this specific example produces only a BIOTA file, the utility\n> can generate multiple files (one per sample type) depending on the\n> content of the source dataset. This reflects the NetCDF4 file\n> structure, where each MARIS sample type is stored as a separate group\n> within the file.\n\n## Development\n\nThe MARIS NetCDF template is generated from `nbs/files/cdl/maris.cdl`\nCommon Data Language (CDL) file as defined by\n[Unidata](https://docs.unidata.ucar.edu/). To generate the MARIS NetCDF\ntemplate `nbs/files/nc/maris-template.nc`, install the\n[NetCDF-C](https://pjbartlein.github.io/REarthSysSci/install_netCDF.html)\nutilities, once in `Marisco` home directory, run:\n\n``` console\nncgen -4 -o nc/maris-template.nc cdl/maris.cdl\n```\n",
"bugtrack_url": null,
"license": "Apache Software License 2.0",
"summary": "MARIS companion package and tutorials",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/franckalbinet/marisco"
},
"split_keywords": [
"nbdev",
"jupyter",
"notebook",
"python",
"netcdf",
"marine",
"radioactivity",
"data"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "2ef30e196e5abcccf5267533ac358ade046e2a3fc8060e7c3309662728e5447e",
"md5": "afe6f61a2a980f5b47e7ae2cd4f89e0e",
"sha256": "3dd17bfc2e8ca690b7708267df5dd787edb96e820a68d0b01db013d9abb3c88e"
},
"downloads": -1,
"filename": "marisco-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "afe6f61a2a980f5b47e7ae2cd4f89e0e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 69879,
"upload_time": "2025-02-18T10:30:49",
"upload_time_iso_8601": "2025-02-18T10:30:49.238634Z",
"url": "https://files.pythonhosted.org/packages/2e/f3/0e196e5abcccf5267533ac358ade046e2a3fc8060e7c3309662728e5447e/marisco-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "35bac5c4c5c58da78ba1611e015248145c53ddb736f44792eace12269ef52d8e",
"md5": "277930a80b5ed576c3d92289d9fa1841",
"sha256": "57145141066dee68c8485d92c5b979bfb4232f4876ce824dba369227465907fc"
},
"downloads": -1,
"filename": "marisco-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "277930a80b5ed576c3d92289d9fa1841",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 63997,
"upload_time": "2025-02-18T10:30:51",
"upload_time_iso_8601": "2025-02-18T10:30:51.439900Z",
"url": "https://files.pythonhosted.org/packages/35/ba/c5c4c5c58da78ba1611e015248145c53ddb736f44792eace12269ef52d8e/marisco-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-18 10:30:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "franckalbinet",
"github_project": "marisco",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "marisco"
}
JSON
