# IMAP Data Access Package
This lightweight Python package allows users to download, query, and upload data from the IMAP Science Data Center (SDC).
## Command Line Utility
### To install
```bash
pip install imap-data-access
imap-data-access -h
```
### Query / Search for data
Find all files from the SWE instrument
```bash
$ imap-data-access query --instrument swe
Found [2] matching files
---------------------------------------------------------------------------------------------------------------|
Instrument|Data Level|Descriptor|Start Date|Repointing|Version|Filename |
---------------------------------------------------------------------------------------------------------------|
swe |l0 |sci |20240105 | |v001 |imap_swe_l0_sci_20240105_v001.pkts |
swe |l0 |sci |20240105 | |v001 |imap_swe_l0_sci_20240105_v001.pkts |
---------------------------------------------------------------------------------------------------------------|
```
Find all files during the year 2024 and return the response as raw json
```bash
$ imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json
[{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}]
```
### Download a file
Download a level 0 SWE file on 2024/01/05
```bash
$ imap-data-access download imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts
Successfully downloaded the file to: <IMAP_DATA_DIR>/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts
```
### Upload a file
Upload a l1a file after decoding the l0 CCSDS ".pkts" file
```bash
$ imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf
Successfully uploaded the file to the IMAP SDC
```
## Importing as a package
```python
import imap_data_access
# Search for files
results = imap_data_access.query(instrument="mag", data_level="l0")
# results is a list of dictionaries
# [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105','version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}]
# Download a file that was returned from the search
imap_data_access.download("imap/mag/l0/2024/01/imap_mag_l0_raw_202040101_v001.pkts")
# Upload a calibration file that exists locally
imap_data_access.upload("imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf")
```
## Configuration
### Data directory
The folder structure for data files within the IMAP SDC is rigidly
defined, so the data access will mimic that structure to make sure
all data is stored in the same heirarchical structure as the SDC.
This will enable seamless transition between a user's local system
and the SDC.
A user's root data location can be specified as an environment
variable ``IMAP_DATA_DIR`` or through a configuration dictionary
within the package itself ``imap_data_access.config["DATA_DIR"]``.
If the ``IMAP_DATA_DIR`` variable is not set, the program defaults
to the user's current working directory + ``data/``.
The following is the directory structure the IMAP SDC uses.
```text
<IMAP_DATA_DIR>/
imap/
<instrument>/
<data_level>/
<year>/
<month>/
<filename>
```
for example, with ``IMAP_DATA_DIR=/data``:
```text
/data/
imap/
swe/
l0/
2024/
01/
imap_swe_l0_sci_20240105_v001.pkts
```
### Data Access URL
To change the default URL that the package accesses, you can set
the environment variable ``IMAP_DATA_ACCESS_URL`` or within the
package ``imap_data_access.config["DATA_ACCESS_URL"]``. The default
is the development server ``https://api.dev.imap-mission.com``.
## Troubleshooting
### Network issues
#### SSL
If you encounter SSL errors similar to the following:
```text
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
```
That generally means the Python environment you're using is not finding your system's root
certificates properly. This means you need to tell Python how to find those certificates
with the following potential solutions.
1. **Upgrade the certifi package**
```bash
pip install --upgrade certifi
```
2. **Install system certificates**
Depending on the Python version you installed the program with the command will look something like this:
```bash
/Applications/Python\ 3.10/Install\ Certificates.command
```
#### HTTP Error 502: Bad Gateway
This could mean that the service is temporarily down. If you
continue to encounter this, reach out to the IMAP SDC at
<imap-sdc@lasp.colorado.edu>.
#### FileNotFoundError
This could mean that the local data directory is not set
up with the same paths as the SDC. See the [data directory](#data-directory)
section for an example of how to set this up.
## File Validation
This package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for
use by other packages to create filepaths and filenames that follow the IMAP SDC conventions.
To use this class, use `imap_data_access.ScienceFilepath`.
Usage:
```python
science_file = imap_data_access.ScienceFilePath("imap_swe_l0_sci_20240101_v001.pkts")
# Filepath = /imap/swe/l0/2024/01/imap_swe_l0_sci_20240101_v001.pkts
filepath = science_file.construct_file_path()
```
Raw data
{
"_id": null,
"home_page": null,
"name": "imap-data-access",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "IMAP, SDC, SOC, SDS, Science Operations",
"author": null,
"author_email": "IMAP SDC Developers <imap-sdc@lists.lasp.colorado.edu>",
"download_url": "https://files.pythonhosted.org/packages/73/30/05a4f631f2e353ae1bf9eb47898dbd0446be22676a1d6f4f23f48eb065a0/imap_data_access-0.6.1.tar.gz",
"platform": null,
"description": "# IMAP Data Access Package\n\nThis lightweight Python package allows users to download, query, and upload data from the IMAP Science Data Center (SDC).\n\n## Command Line Utility\n\n### To install\n\n```bash\npip install imap-data-access\nimap-data-access -h\n```\n\n### Query / Search for data\n\nFind all files from the SWE instrument\n\n```bash\n$ imap-data-access query --instrument swe\nFound [2] matching files\n---------------------------------------------------------------------------------------------------------------|\nInstrument|Data Level|Descriptor|Start Date|Repointing|Version|Filename |\n---------------------------------------------------------------------------------------------------------------|\nswe |l0 |sci |20240105 | |v001 |imap_swe_l0_sci_20240105_v001.pkts |\nswe |l0 |sci |20240105 | |v001 |imap_swe_l0_sci_20240105_v001.pkts |\n---------------------------------------------------------------------------------------------------------------|\n```\n\nFind all files during the year 2024 and return the response as raw json\n\n```bash\n$ imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json\n[{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}]\n```\n\n### Download a file\n\nDownload a level 0 SWE file on 2024/01/05\n\n```bash\n$ imap-data-access download imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts\nSuccessfully downloaded the file to: <IMAP_DATA_DIR>/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts\n```\n\n### Upload a file\n\nUpload a l1a file after decoding the l0 CCSDS \".pkts\" file\n\n```bash\n$ imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf\nSuccessfully uploaded the file to the IMAP SDC\n```\n\n## Importing as a package\n\n```python\nimport imap_data_access\n\n# Search for files\nresults = imap_data_access.query(instrument=\"mag\", data_level=\"l0\")\n# results is a list of dictionaries\n# [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105','version': 'v001', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}]\n\n# Download a file that was returned from the search\nimap_data_access.download(\"imap/mag/l0/2024/01/imap_mag_l0_raw_202040101_v001.pkts\")\n\n# Upload a calibration file that exists locally\nimap_data_access.upload(\"imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf\")\n```\n\n## Configuration\n\n### Data directory\n\nThe folder structure for data files within the IMAP SDC is rigidly\ndefined, so the data access will mimic that structure to make sure\nall data is stored in the same heirarchical structure as the SDC.\nThis will enable seamless transition between a user's local system\nand the SDC.\n\nA user's root data location can be specified as an environment\nvariable ``IMAP_DATA_DIR`` or through a configuration dictionary\nwithin the package itself ``imap_data_access.config[\"DATA_DIR\"]``.\nIf the ``IMAP_DATA_DIR`` variable is not set, the program defaults\nto the user's current working directory + ``data/``.\n\nThe following is the directory structure the IMAP SDC uses.\n\n```text\n<IMAP_DATA_DIR>/\n imap/\n <instrument>/\n <data_level>/\n <year>/\n <month>/\n <filename>\n```\n\nfor example, with ``IMAP_DATA_DIR=/data``:\n\n```text\n/data/\n imap/\n swe/\n l0/\n 2024/\n 01/\n imap_swe_l0_sci_20240105_v001.pkts\n```\n\n### Data Access URL\n\nTo change the default URL that the package accesses, you can set\nthe environment variable ``IMAP_DATA_ACCESS_URL`` or within the\npackage ``imap_data_access.config[\"DATA_ACCESS_URL\"]``. The default\nis the development server ``https://api.dev.imap-mission.com``.\n\n## Troubleshooting\n\n### Network issues\n\n#### SSL\n\nIf you encounter SSL errors similar to the following:\n\n```text\nurllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>\n```\n\nThat generally means the Python environment you're using is not finding your system's root\ncertificates properly. This means you need to tell Python how to find those certificates\nwith the following potential solutions.\n\n1. **Upgrade the certifi package**\n\n ```bash\n pip install --upgrade certifi\n ```\n\n2. **Install system certificates**\n Depending on the Python version you installed the program with the command will look something like this:\n\n ```bash\n /Applications/Python\\ 3.10/Install\\ Certificates.command\n ```\n\n#### HTTP Error 502: Bad Gateway\n\nThis could mean that the service is temporarily down. If you\ncontinue to encounter this, reach out to the IMAP SDC at\n<imap-sdc@lasp.colorado.edu>.\n\n#### FileNotFoundError\n\nThis could mean that the local data directory is not set\nup with the same paths as the SDC. See the [data directory](#data-directory)\nsection for an example of how to set this up.\n\n## File Validation\n\nThis package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for\nuse by other packages to create filepaths and filenames that follow the IMAP SDC conventions.\n\nTo use this class, use `imap_data_access.ScienceFilepath`.\n\nUsage:\n\n```python\n\nscience_file = imap_data_access.ScienceFilePath(\"imap_swe_l0_sci_20240101_v001.pkts\")\n\n# Filepath = /imap/swe/l0/2024/01/imap_swe_l0_sci_20240101_v001.pkts\nfilepath = science_file.construct_file_path()\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "IMAP SDC Data Access",
"version": "0.6.1",
"project_urls": {
"homepage": "https://github.com/IMAP-Science-Operations-Center",
"repository": "https://github.com/IMAP-Science-Operations-Center/imap-data-access"
},
"split_keywords": [
"imap",
" sdc",
" soc",
" sds",
" science operations"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "733005a4f631f2e353ae1bf9eb47898dbd0446be22676a1d6f4f23f48eb065a0",
"md5": "180acecde20e00b90d6ab3b32b727fad",
"sha256": "ad36efbce05cb829af843e7562cddb648d3cd4cbfc718390d4b01d6a009bf7de"
},
"downloads": -1,
"filename": "imap_data_access-0.6.1.tar.gz",
"has_sig": false,
"md5_digest": "180acecde20e00b90d6ab3b32b727fad",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 19365,
"upload_time": "2024-05-08T21:34:11",
"upload_time_iso_8601": "2024-05-08T21:34:11.966050Z",
"url": "https://files.pythonhosted.org/packages/73/30/05a4f631f2e353ae1bf9eb47898dbd0446be22676a1d6f4f23f48eb065a0/imap_data_access-0.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-08 21:34:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "IMAP-Science-Operations-Center",
"github_project": "imap-data-access",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "imap-data-access"
}
JSON
