<!-- markdownlint-disable line-length-->
# Dicom Fixer
## Overview
### Summary
This gear is responsible for reporting on the Dicom files data elements and optionally
fixing/enhancing the problematic ones and generating a new archive.
### Cite
*License*: *MIT*
### Classification
*Category:* *Converter*
*Gear Level:*
- [ ] Project
- [ ] Subject
- [ ] Session
- [X] Acquisition
- [ ] Analysis
----
[[*TOC*]]
----
### Inputs
- __dicom__: Input dicom
- *dicom*
- __Name__: *dicom*
- __Type__: *file*
- __Optional__: *False*
- __Classification__: *DICOM file*
- __Description__: *Input DICOM to be fixed*
### Config
- *debug*
- __Name__: *debug*
- __Type__: *boolean*
- __Description__: *Include debug statements in output.*
- __Default__: `False`
- *dicom-standard*
- __Name__: *dicom-standard*
- __Type__: *string*
- __Description__: *Specify which DICOM standard edition to use. `local` refers to the locally-saved edition for faster processing. `current` fetches the most up-to-date edition at runtime.*
- __Default__: `"local"`
- *force_decompress*
- __Name__: *force_decompress*
- __Type__: *boolean*
- __Description__: *Expert option: Force standardize_transfer_syntax fix even if filesize may be too big for available memory. WARNING: Choosing this option may result in the gear failing due to being out of memory.*
- __Default__: `False`
- *standardize_transfer_syntax*
- __Name__: *standardize_transfer_syntax*
- __Type__: *boolean*
- __Description__: *Whether or not to change `TransferSyntaxUID` to ExplicitVRLittleEndian.*
- __Default__: `True`
- *strict-validation*
- __Name__: *strict-validation*
- __Type__: *boolean*
- __Description__: *Enforce strict DICOM validation if true; else, allow python-parsable values that may not meet DICOM standard.*
- __Default__: `True`
- *tag*
- __Name__: *tag*
- __Type__: *string*
- __Description__: *The tag to be added on input file upon run completion.*
- __Default__: `"dicom-fixer"`
- *unique*
- __Name__: *unique*
- __Type__: *boolean*
- __Description__: *Enforce DICOM uniqueness by SOPInstanceUID or file hash. Remove duplicate DICOMs.*
- __Default__: `True`
- *zip-single-dicom*
- __Name__: *zip-single-dicom*
- __Type__: *string*
- __Description__: *Output a single DICOM as zip (.dcm.zip) instead of a DICOM (.dcm) or match input.*
- __Default__: `"match"`
### Metadata
This gear identifies and fixes invalid DICOM tags of the input file. Additionally, dicom-fixer will add a QC result titled `fixed` to either the input file if no output is written, or the output file if there is an output. The state will be either `PASS` or
`FAIL`. For a `PASS` result, any fixing events will be under the `events` key.
### Prerequisites
#### Prerequisite Gear Runs
1. __*Splitter*__
- Level: Acquisition
- Note: If multiple SeriesInstanceUIDs are present within the DICOM collection, Splitter *must* be run before Dicom-Fixer.
#### Prerequisite Files
No prerequisite files.
#### Prerequisite Metadata
No prerequisite metadata.
## Usage
### Description
This gear checks the input DICOM's tags and makes corrections as needed, whether configured to adhere to DICOM Standard (when `strict-validation=True`) or configured to only check for python parsability (when `strict-validation=False`).
This gear utilizes the `tracker` functionality of `RawDataElements`
in [fw-file](https://gitlab.com/flywheel-io/tools/lib/fw-file.git). Namely, it houses
a collection of default and custom fixers that are applied to `RawDataElements` on
read through `pydicom`.
Dicom fixer will also decompress compressed TransferSyntaxes so that no issues are met
on the platform downstream.
### Standardize transfer syntax
The `standardize_transfer_syntax` option supports the following Transfer Syntaxes:
| Name | UID | Supported |
|------------------------------------|------------------------|--------------------|
| Explicit VR Little Endian | 1.2.840.10008.1.2.1 | :white_check_mark: |
| Implicit VR Little Endian | 1.2.840.10008.1.2 | :white_check_mark: |
| Explicit VR Big Endian | 1.2.840.10008.1.2.2 | :white_check_mark: |
| Deflated Explicit VR Little Endian | 1.2.840.10008.1.2.1.99 | :white_check_mark: |
| RLE Lossless | 1.2.840.10008.1.2.5 | :white_check_mark: |
| JPEG Baseline (Process 1) | 1.2.840.10008.1.2.4.50 | :white_check_mark: |
| JPEG Extended (Process 2 and 4) | 1.2.840.10008.1.2.4.51 | :white_check_mark: |
| JPEG Lossless (Process 14) | 1.2.840.10008.1.2.4.57 | :white_check_mark: |
| JPEG Lossless (Process 14, SV1) | 1.2.840.10008.1.2.4.70 | :white_check_mark: |
| JPEG LS Lossless | 1.2.840.10008.1.2.4.80 | :white_check_mark: |
| JPEG LS Lossy | 1.2.840.10008.1.2.4.81 | :white_check_mark: |
| JPEG2000 Lossless | 1.2.840.10008.1.2.4.90 | :white_check_mark: |
| JPEG2000 | 1.2.840.10008.1.2.4.91 | :white_check_mark: |
| JPEG2000 Multi-component Lossless | 1.2.840.10008.1.2.4.92 | :x: |
| JPEG2000 Multi-component | 1.2.840.10008.1.2.4.93 | :x: |
All transfer syntax decompressing is done with pydicom using either `numpy` and
`GDCM`, `JPEG-LS`, `Pillow`, or `pylibjpeg`, see more info on pydicom's docs on
[supported transfer syntaxes](https://pydicom.github.io/pydicom/stable/old/image_data_handlers.html)
### Color space conversion
For color files, after decompression (usually jpeg) the color space will be converted to RGB for better downstream support. Currently color space conversion is supported for the following PhotometricInterpretations:
- YBR_FULL_422
- YBR_FULL
- PALETTE COLOR
### File Specifications
#### Input
The input of this gear is a DICOM file.
### Workflow
```mermaid
graph LR;
A[dicom]:::input --> C;
C[Upload] --> D[Acquisition];
D:::container --> E((Gear));
E:::gear --> F[Input replaced with fixed DICOM if fixes applied];
classDef container fill:#57d,color:#fff
classDef input fill:#7a9,color:#fff
classDef gear fill:#659,color:#fff
```
Description of workflow:
1. Upload file to container
1. Select file as input to gear
1. Configure gear as needed
1. If fixes are required and gear passes, gear replaces file with an updated version and QC is marked as `PASS`. If fixes are required and gear does not pass, QC is marked as `FAIL`. If no fixes are identified, input file is retained and marked as `PASS`.
### Fixers
The following fixes are applied:
#### General fixes from [fw-file](https://gitlab.com/flywheel-io/tools/lib/fw-file)
- Fix VR for SpecificCharacterSet and surplus SequenceDelimitationItem tags.
- Replace VR=None with VR found in the public or private dictionaries.
- Replace VR='UN' with VR found in the public or private dictionaries.
- Replace invalid \ characters with _ in string VR values of VM=1.
- Crop text VRs which are too long.
- Fix an invalid UID. Determine if UID is semi-valid (e.g. composed of a minimum 5 nodes
including invalid node starting with 0), if semi-valid, generates a new UID with the
semi-valid UID as entropy source (deterministic), else, generates a new UID.
- Fix date times tags. Attempt to parse an invalid date and format correctly.
- Fix AS strings. Ensure one of D,W,M,Y is at end of the string, and pad to 4
characters. If no time quantifier is present, assume Years.
- Fix number strings. Fix DS (Decimal String) and IS (Integer String) number strings by
removing invalid characters and truncate floats in IS VR.
- Fix invalid character. Attempt to remove non-printable characters from byte decoding.
- Fix invalid VR value. Try to fix an invalid value for the given VR.
- Fix LUT Descriptor tags.
#### Custom fixes
- Fix incorrect units. Correct MagneticFieldStrength from milli-Tesla to Tesla.
- Remove file in archive that are not likely DICOM (defined as having at least 2
public dicom tags outside the file_meta (0000, 0002) group).
- Optionally, sets the TransferSyntaxUID to ExplicitVRLittleEndian.
### Output
#### File
DICOM-fixer will either output a fixed file to overwrite the input or will output
nothing
#### QC
DICOM-fixer will add a QC result titled `fixed` to either the input file if no output is
written, or the output file if there is an output. The state will be either `PASS` or
`FAIL`:
- `PASS` if either fixes were attempted and write was successful, or if no fixes were
attempted.
- `FAIL` if fixes were attempted but write was not successful.
For a `PASS` result, any fixing events will be under the `events` key.
### Use Cases
#### Use Case 1: Invalid DICOM tags, strict validation
__*DICOM has invalid tag values and needs to adhere to DICOM Standard*__:
For DICOMs that must adhere to DICOM Standard, the `strict-validation` config option should be set to `True`. `dicom-standard` can be set to `"local"` (default) for faster processing with the locally-saved DICOM Standard, or set to `"current"` to retrieve the most recent DICOM Standard at runtime.
#### Use Case 2: Invalid DICOM tags, non-strict validation
__*DICOM has invalid tag values, but information retention is preferred over strict adherance to DICOM Standard*__:
For situations where information retention is preferred over strict DICOM standard adherance, the `strict-validation` config option should be set to `False` (default). This tells the gear to keep tag values unless they are not python parsable.
### Logging
The gear logs events as it processes the input DICOM. If fixes are required, the gear log notates these as they are identified. The gear then logs whether or not output is written and reason for writing output. The `.metadata.json` created by the gear is logged, including the parent file, gear configuration, QC results, and file tags. When `debug` is set to `True`, debugging-level statements are logged.
## FAQ
[FAQ.md](FAQ.md)
## Contributing
For more information about how to get started contributing to that gear,
checkout [CONTRIBUTING.md](CONTRIBUTING.md).
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/flywheel-io/flywheel-apps/dicom-fixer",
"name": "fw-gear-dicom-fixer",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "Flywheel, Gears, Dicom, Fixer",
"author": "Flywheel",
"author_email": "support@flywheel.io",
"download_url": null,
"platform": null,
"description": "<!-- markdownlint-disable line-length-->\n# Dicom Fixer\n\n## Overview\n\n### Summary\n\nThis gear is responsible for reporting on the Dicom files data elements and optionally\nfixing/enhancing the problematic ones and generating a new archive. \n\n### Cite\n\n*License*: *MIT*\n\n### Classification\n\n*Category:* *Converter*\n\n*Gear Level:*\n\n- [ ] Project\n- [ ] Subject\n- [ ] Session\n- [X] Acquisition\n- [ ] Analysis\n\n----\n\n[[*TOC*]]\n\n----\n\n### Inputs\n\n- __dicom__: Input dicom\n\n- *dicom*\n - __Name__: *dicom*\n - __Type__: *file*\n - __Optional__: *False*\n - __Classification__: *DICOM file*\n - __Description__: *Input DICOM to be fixed*\n\n### Config\n\n- *debug*\n - __Name__: *debug*\n - __Type__: *boolean*\n - __Description__: *Include debug statements in output.*\n - __Default__: `False`\n- *dicom-standard*\n - __Name__: *dicom-standard*\n - __Type__: *string*\n - __Description__: *Specify which DICOM standard edition to use. `local` refers to the locally-saved edition for faster processing. `current` fetches the most up-to-date edition at runtime.*\n - __Default__: `\"local\"`\n- *force_decompress*\n - __Name__: *force_decompress*\n - __Type__: *boolean*\n - __Description__: *Expert option: Force standardize_transfer_syntax fix even if filesize may be too big for available memory. WARNING: Choosing this option may result in the gear failing due to being out of memory.*\n - __Default__: `False`\n- *standardize_transfer_syntax*\n - __Name__: *standardize_transfer_syntax*\n - __Type__: *boolean*\n - __Description__: *Whether or not to change `TransferSyntaxUID` to ExplicitVRLittleEndian.*\n - __Default__: `True`\n- *strict-validation*\n - __Name__: *strict-validation*\n - __Type__: *boolean*\n - __Description__: *Enforce strict DICOM validation if true; else, allow python-parsable values that may not meet DICOM standard.*\n - __Default__: `True`\n- *tag*\n - __Name__: *tag*\n - __Type__: *string*\n - __Description__: *The tag to be added on input file upon run completion.*\n - __Default__: `\"dicom-fixer\"`\n- *unique*\n - __Name__: *unique*\n - __Type__: *boolean*\n - __Description__: *Enforce DICOM uniqueness by SOPInstanceUID or file hash. Remove duplicate DICOMs.*\n - __Default__: `True`\n- *zip-single-dicom*\n - __Name__: *zip-single-dicom*\n - __Type__: *string*\n - __Description__: *Output a single DICOM as zip (.dcm.zip) instead of a DICOM (.dcm) or match input.*\n - __Default__: `\"match\"`\n\n### Metadata\n\nThis gear identifies and fixes invalid DICOM tags of the input file. Additionally, dicom-fixer will add a QC result titled `fixed` to either the input file if no output is written, or the output file if there is an output. The state will be either `PASS` or\n`FAIL`. For a `PASS` result, any fixing events will be under the `events` key.\n\n### Prerequisites\n\n#### Prerequisite Gear Runs\n\n1. __*Splitter*__\n - Level: Acquisition\n - Note: If multiple SeriesInstanceUIDs are present within the DICOM collection, Splitter *must* be run before Dicom-Fixer.\n\n#### Prerequisite Files\n\nNo prerequisite files.\n\n#### Prerequisite Metadata\n\nNo prerequisite metadata.\n\n## Usage\n\n### Description\n\nThis gear checks the input DICOM's tags and makes corrections as needed, whether configured to adhere to DICOM Standard (when `strict-validation=True`) or configured to only check for python parsability (when `strict-validation=False`).\n\nThis gear utilizes the `tracker` functionality of `RawDataElements`\nin [fw-file](https://gitlab.com/flywheel-io/tools/lib/fw-file.git). Namely, it houses\na collection of default and custom fixers that are applied to `RawDataElements` on\nread through `pydicom`.\n\nDicom fixer will also decompress compressed TransferSyntaxes so that no issues are met\non the platform downstream.\n\n### Standardize transfer syntax\n\nThe `standardize_transfer_syntax` option supports the following Transfer Syntaxes:\n\n| Name | UID | Supported |\n|------------------------------------|------------------------|--------------------|\n| Explicit VR Little Endian | 1.2.840.10008.1.2.1 | :white_check_mark: |\n| Implicit VR Little Endian | 1.2.840.10008.1.2 | :white_check_mark: |\n| Explicit VR Big Endian | 1.2.840.10008.1.2.2 | :white_check_mark: |\n| Deflated Explicit VR Little Endian | 1.2.840.10008.1.2.1.99 | :white_check_mark: |\n| RLE Lossless | 1.2.840.10008.1.2.5 | :white_check_mark: |\n| JPEG Baseline (Process 1) | 1.2.840.10008.1.2.4.50 | :white_check_mark: |\n| JPEG Extended (Process 2 and 4) | 1.2.840.10008.1.2.4.51 | :white_check_mark: |\n| JPEG Lossless (Process 14) | 1.2.840.10008.1.2.4.57 | :white_check_mark: |\n| JPEG Lossless (Process 14, SV1) | 1.2.840.10008.1.2.4.70 | :white_check_mark: |\n| JPEG LS Lossless | 1.2.840.10008.1.2.4.80 | :white_check_mark: |\n| JPEG LS Lossy | 1.2.840.10008.1.2.4.81 | :white_check_mark: |\n| JPEG2000 Lossless | 1.2.840.10008.1.2.4.90 | :white_check_mark: |\n| JPEG2000 | 1.2.840.10008.1.2.4.91 | :white_check_mark: |\n| JPEG2000 Multi-component Lossless | 1.2.840.10008.1.2.4.92 | :x: |\n| JPEG2000 Multi-component | 1.2.840.10008.1.2.4.93 | :x: |\n\nAll transfer syntax decompressing is done with pydicom using either `numpy` and\n`GDCM`, `JPEG-LS`, `Pillow`, or `pylibjpeg`, see more info on pydicom's docs on\n[supported transfer syntaxes](https://pydicom.github.io/pydicom/stable/old/image_data_handlers.html)\n\n### Color space conversion\n\nFor color files, after decompression (usually jpeg) the color space will be converted to RGB for better downstream support. Currently color space conversion is supported for the following PhotometricInterpretations:\n\n- YBR_FULL_422\n- YBR_FULL\n- PALETTE COLOR\n\n### File Specifications\n\n#### Input\n\nThe input of this gear is a DICOM file.\n\n### Workflow\n\n```mermaid\ngraph LR;\n A[dicom]:::input --> C;\n C[Upload] --> D[Acquisition];\n D:::container --> E((Gear));\n\n E:::gear --> F[Input replaced with fixed DICOM if fixes applied];\n \n classDef container fill:#57d,color:#fff\n classDef input fill:#7a9,color:#fff\n classDef gear fill:#659,color:#fff\n\n```\n\nDescription of workflow:\n\n1. Upload file to container\n1. Select file as input to gear\n1. Configure gear as needed\n1. If fixes are required and gear passes, gear replaces file with an updated version and QC is marked as `PASS`. If fixes are required and gear does not pass, QC is marked as `FAIL`. If no fixes are identified, input file is retained and marked as `PASS`.\n\n### Fixers\n\nThe following fixes are applied:\n\n#### General fixes from [fw-file](https://gitlab.com/flywheel-io/tools/lib/fw-file)\n\n- Fix VR for SpecificCharacterSet and surplus SequenceDelimitationItem tags.\n- Replace VR=None with VR found in the public or private dictionaries.\n- Replace VR='UN' with VR found in the public or private dictionaries.\n- Replace invalid \\ characters with _ in string VR values of VM=1.\n- Crop text VRs which are too long.\n- Fix an invalid UID. Determine if UID is semi-valid (e.g. composed of a minimum 5 nodes\n including invalid node starting with 0), if semi-valid, generates a new UID with the\n semi-valid UID as entropy source (deterministic), else, generates a new UID.\n- Fix date times tags. Attempt to parse an invalid date and format correctly.\n- Fix AS strings. Ensure one of D,W,M,Y is at end of the string, and pad to 4\n characters. If no time quantifier is present, assume Years.\n- Fix number strings. Fix DS (Decimal String) and IS (Integer String) number strings by\n removing invalid characters and truncate floats in IS VR.\n- Fix invalid character. Attempt to remove non-printable characters from byte decoding.\n- Fix invalid VR value. Try to fix an invalid value for the given VR.\n- Fix LUT Descriptor tags.\n \n#### Custom fixes\n\n- Fix incorrect units. Correct MagneticFieldStrength from milli-Tesla to Tesla.\n- Remove file in archive that are not likely DICOM (defined as having at least 2\n public dicom tags outside the file_meta (0000, 0002) group).\n- Optionally, sets the TransferSyntaxUID to ExplicitVRLittleEndian.\n\n### Output\n\n#### File\n\nDICOM-fixer will either output a fixed file to overwrite the input or will output\nnothing\n\n#### QC\n\nDICOM-fixer will add a QC result titled `fixed` to either the input file if no output is\nwritten, or the output file if there is an output. The state will be either `PASS` or\n`FAIL`:\n\n- `PASS` if either fixes were attempted and write was successful, or if no fixes were\nattempted. \n- `FAIL` if fixes were attempted but write was not successful.\n\nFor a `PASS` result, any fixing events will be under the `events` key.\n\n### Use Cases\n\n#### Use Case 1: Invalid DICOM tags, strict validation\n\n__*DICOM has invalid tag values and needs to adhere to DICOM Standard*__:\n\nFor DICOMs that must adhere to DICOM Standard, the `strict-validation` config option should be set to `True`. `dicom-standard` can be set to `\"local\"` (default) for faster processing with the locally-saved DICOM Standard, or set to `\"current\"` to retrieve the most recent DICOM Standard at runtime.\n\n#### Use Case 2: Invalid DICOM tags, non-strict validation\n\n__*DICOM has invalid tag values, but information retention is preferred over strict adherance to DICOM Standard*__:\n\nFor situations where information retention is preferred over strict DICOM standard adherance, the `strict-validation` config option should be set to `False` (default). This tells the gear to keep tag values unless they are not python parsable.\n\n### Logging\n\nThe gear logs events as it processes the input DICOM. If fixes are required, the gear log notates these as they are identified. The gear then logs whether or not output is written and reason for writing output. The `.metadata.json` created by the gear is logged, including the parent file, gear configuration, QC results, and file tags. When `debug` is set to `True`, debugging-level statements are logged.\n\n## FAQ\n\n[FAQ.md](FAQ.md)\n\n## Contributing\n\nFor more information about how to get started contributing to that gear,\ncheckout [CONTRIBUTING.md](CONTRIBUTING.md).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "This gear reports on the Dicom files data elements and optionally fixes or enhances the problematic ones, generating a new archive.",
"version": "0.9.6",
"project_urls": {
"Homepage": "https://gitlab.com/flywheel-io/flywheel-apps/dicom-fixer",
"Repository": "https://gitlab.com/flywheel-io/flywheel-apps/dicom-fixer"
},
"split_keywords": [
"flywheel",
" gears",
" dicom",
" fixer"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2bfdcfe660d5fcfeaf324449ba314c4c7fb4789a7183bf34b70ac4f24c6868b8",
"md5": "5bc221dbac4a2daf0408426c1c5e29fd",
"sha256": "aba6d9cabfdafcb45f74b6456c64d87daf094f6d6e95262ed5410973f6d4f957"
},
"downloads": -1,
"filename": "fw_gear_dicom_fixer-0.9.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5bc221dbac4a2daf0408426c1c5e29fd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 10899657,
"upload_time": "2024-04-02T21:52:15",
"upload_time_iso_8601": "2024-04-02T21:52:15.679945Z",
"url": "https://files.pythonhosted.org/packages/2b/fd/cfe660d5fcfeaf324449ba314c4c7fb4789a7183bf34b70ac4f24c6868b8/fw_gear_dicom_fixer-0.9.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-02 21:52:15",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "flywheel-io",
"gitlab_project": "flywheel-apps",
"lcname": "fw-gear-dicom-fixer"
}
JSON