# pymusepipe
WARNING: Doc need to be updated since many major changes. Coming asap.
This is a Python wrapper for the MUSE/VLT pipeline recipes. This runs
using the esorex command lines (in parallel, using by default likwid, although
other options are available). This wrapper is meant to reduce MUSE OBs automatically,
after dumping the fits files in a given folder.
This package includes an alignment module which may be useful, independently given
a set of images to align and assuming a reference image is provided. It can spit out
an "OFFSET_LIST" MUSE compatible fits table including the flux scaling factors.
pymusepipe is also made for multi-pointing mosaics and multi-targets surveys
as it will process targets automatically when provided a specific dictionary
of which target and which pointings to consider.
## 1 - Getting Started
Here are some basic instructions to get you going with pymusepipe.
### Prerequisites
pymusepipe uses Python 3. It is not compatible with python 2.7. If you are
still using python 2.7 I would recommend you to switch asap as it won't be
supported much longer anyway.
pymusepipe needs a number of standard modules/packages including:
* **numpy**
* **scipy** for the alignment module
* **matplotlib** if you wish to make some plots (e.g. via the check_pipe module)
* **astropy** as a general provider of many useful functions and structures, most importantly the astropy Tables.
The convolution package of pymusepipe (allowing full convolution of a datacube) uses
**pypher** which is thus a requirement for that module to work.
This also includes a heavy-based usage of **mpdaf**, which is a super useful package
developed by the MUSE GTO-CRAL Team, to process and analyse datacubes, and more specifically
MUSE cubes, images and spectra.
### Installing
You can install this package via Pypi via a simple:
```buildoutcfg
pip install pymusepipe
```
You can obviously also install it by cloning it from github, or
downloading the source (from github) and do something like:
```
python setup.py develop
```
I advise to use the "develop" option as it actually does not copy
the files in your system but just create a link.
In that way you can easily update the source software without
reinstalling it. The link will directly use the source which has been udpated.
The other option is to use the standard "install" option:
```
python setup.py install
```
## 2 - Basic Usage - Dealing with OBs, individually
The pymusepipe wrapper is meant to provide the user with a simple way to
run the MUSE pipeline.
Only 3 steps are needed: preparing the data (download), preparing the
configuration files (templates are provided) which should take a couple of minutes,
and using the code (a few lines at most). The package has many other hidden functionalities
which I'll document as soon as I can.
I recommend to use ```Ipython``` as an environment, possibly via
a ```screen``` which would allow you to disconnect from the server that
actually runs the commands. Jupyter notebooks may also be very handy
although I would not necessarily recommend them for this specific purpose as
the running of the pipeline may take hours.
### Step 1 = Preparing your data
The first thing to do is to prepare the folder structure to welcome your
MUSE datasets. This is actually rather simple.
Imagine you have
* a target or field named "NGC1000" (just for the sake of using an example).
* several pointings (each pointing can represent a set of OBs which have
roughly the same coordinates, best is that they have a very significant
overlap): 1, 2, and 3. Note that this 'pointing' structure may not reflect your
own observational strategy. There are cases where one OB includes different pointings.
What is important here is that each 'pointing' is either a full single OB, or several
OBs with the same pointing.
Then under your data root folder <my_data_folder> create the following folder structure:
```
<my_data_folder>/NGC1000
./P01
./Raw
./P02
./Raw
./P03
./Raw
```
Each pointing/OB has a "Raw" folder.
The next step is to download your MUSE data from the ESO web site,
and put all the raw files (fits/fits.gz) into each individual "Raw" folder,
associated with the right pointing.
### Step 2 = Preparing your configuration files
pymusepipe only needs 2 configurations ASCII files: 1 for the calibration
(calib_tables.dic) and one for the root folders (rc.dic).
**Examples** of such files are provided in the "config_templates"
folder of the pymusepipe package.
### Step 3 = Running the pipeline
The pipeline is meant to be run automatically from a given python
structure. This should thus take the user only a few lines of codes,
including one to import pymusepipe, a couple to define the names of
the configuration files, one to initialise the musepipe python structure,
and one to launch the recipes. Here is an example of how this should look:
```buildoutcfg
# Import the modules
import pymusepipe as pmp
from pymusepipe import musepipe
# define the names of the configuration files
rcfile = "/my_data/MUSE/Config/rc.dic"
calfile = "my_data_MUSE/Config/calib_tables.dic"
# Initialisation of the python - MusePipe Class - structure
mypipe = musepipe.MusePipe(targetname="NGCXXXX", pointing=1, rc_filename=rcfile,
cal_filename=calfile, log_filename="NGCXXXX_version01.log",
fakemode=False, overwrite_astropy_table=True,
filter_list="white,Cousins_R",
filter_for_alignment="Cousins_R")
# Launching the pipeline
mypipe.run_recipes()
```
Some explanation may be needed to understand what is happening:
* "targetname": is just the name of the target, used to decided where the data will be
* "pointing": will be used as "P01" for pointing=1, etc.
* "logfile": name of the logging file. Note that this logfile is actually
a shell-like file which can be used to re-run the pipeline one step at a time. Note that
there will be also 2 more files created using that name <logfile_name>, namely:
a file named <logfile_name>.out and <logfile_name>.err which will
contain the full output of the commands, and the error messages (stdout, stderr, resp.).
* "fakemode": you can set this to True if you just wish to initialise
things without actually running any recipes. The pipeline will only set
things up but if you run any recipes will only "fake" them (not launch any
esorex command, only spitting the code out)
* "filter_list": list of filter names to use to reconstruct images
when building up cubes. This should be part of the filter_list fits
table provided (see calib_tables config file).
* "filter_for_alignment": specific filter name used for alignment between exposures.
Other options can be useful:
* "musemode": this is by default "WFM_NOAO_N" which is the most often
used MUSE mode. This will filter out exposures not compatible with the
given mode. So please beware.
* "reset_log": will reset the log file. By default it is False, hence
new runs will be appended.
* "overwrite_astropy_table": by default this is False. If True,
new runs will rewrite the Astropy output tables.
* "time_astrometry": by default it is False, meaning the pipeline
will try to detect a GEOMETRY and ASTROMETRY Files delivered with the Raw
files by ESO. If set to True, it will use the time dependent astro/geo files
provided by the GTO Team but you would need to make these available on your system.
Hence I would recommend to keep the default (False).
## 3- Advanced Usage - Targets and Mosaics
## 4- Alignment module
Details to come.
## 5- Further details
Here we provide a bit more details about a few issues regarding
the configuration files or set up parameters.
### Configuration files
#### rc configuration file
It contains 3 lines, with: *musecalib*, *musecalib_time* and *root*.
* *root* provides the root folder for your data. For Target NGCXXXX,
and Pointing 1, the Raw data will be looked for
in *root*/TargetXXXX/P01/Raw (see *Preparing your data* above).
* musecalib should contain the standard MUSE calibration files.
These are usually distributed in the MUSE kit within a
"muse-calib-x.x.x/cal" folder.
* musecalib_time: time dependent geometry and astrometry files
(the correspondence between observing run dates and specific
files are given in the dic_geo_astrowcs_table in musepipe.py).
#### calib_tables configuration file
It contains a series of given fits files which will be used by the pipeline. Most names are self-explantory. That includes:
* *geo_table* and *astro_table*: only used if you don't rely on the default time dependent geometry files (see rc file).
* *badpix_table*, *vignetting_mask*, *std_flux_table*, *extinct_table*, *line_catalog* all usually provided with the MUSE pipeline.
* *filter_list* : name of the fits filter list. This is used in case you wish to provide your own. Note that it needs to follow the MUSE standard for such a table.
### Recipes
Most MUSE pipeline recipes are run while run_all_recipes is launched. This can be changed in the prep_recipes_pipe.py or just scripted. Actually run_all_recipes() is just a function which launches (assuming 'mypipe' is your MusePipe structure, see above):
```
mypipe.run_bias()
mypipe.run_flat()
mypipe.run_wave()
mypipe.run_lsf()
mypipe.run_twilight(illum=illum)
mypipe.run_scibasic_all(illum=illum)
mypipe.run_standard()
mypipe.run_sky(fraction=fraction)
mypipe.run_prep_align()
mypipe.run_align_bypointing()
mypipe.run_align_bygroup()
mypipe.run_scipost()
mypipe.run_scipost(expotype="SKY", offset_list=False, skymethod='none')
mypipe.run_combine_pointing()
```
where "illum" is a boolean (default is True), fraction is 0.8.
Feel free to launch these steps one by one (the order is important as in any data reduction process).
### Structure of the output
#### Folders
The structure of the output is driven by a set of folder names described in init_musepipe.py in a few dictionaries (dic_input_folders, dic_folders, dic_folders_target). You can in principle change the names of the folders themselves, but I would advise against that.
The pipeline will create the folder structure automatically, checking whether the folders exist or not.
#### Log files
2 basic log files are spitted out: one is the Esorex output which will
be stored in the "Esorex_log" folder. The other one will be in the "Log"
folder with the name provided at start: that one is like a shell script
which can be used to rerun things directly via the command line.
In the "Log" folder, there will also be, for each log file, a file ".out" and
one with ".err" extensions, respectively including all the stdout and stderr
messages. This may be useful to trace details in the data reduction and problems.
#### Astropy Tables
Each recipe will trigger the creation of a astropy Table.
These are stored under "Astro_Tables". You can use these to monitor
which files have been processed or used.
#### Python structure
Most of the information you may need is actually stored in the
python "MusePipe" class structure. More details to come.
## Authors
* **Eric Emsellem** [2017-2020], at ESO and CRAL
## License
This project is licensed under the MIT License - see the
[LICENSE](LICENSE) file for details
## Acknowledgments
I would like to thank people who have initially sent me
their code-samples, including Bernd Husemann, Dimitri Gadotti,
Lodovico Coccato, Mark den Brok. I would also like to specifically
and warmly thank Rebecca McElroy who supported me with the MUSE
data reduction at the early stages of the development of this package,
and Francesco Santoro who has significantly contributed
in the testing, debugging of the code (version 1), and proposed dedicated python lines
to be integrated in pymusepipe (e.g., alignment module).
Raw data
{
"_id": null,
"home_page": "https://github.com/emsellem/pymusepipe",
"name": "pymusepipe",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "MUSE,DATAREDUCTION",
"author": "Eric Emsellem",
"author_email": "eric.emsellem@eso.org",
"download_url": "https://files.pythonhosted.org/packages/64/f0/9ba2f435596929c4799cbd521faaa298f40951b489face4267058412b64b/pymusepipe-2.28.0.tar.gz",
"platform": "any",
"description": "# pymusepipe\n\nWARNING: Doc need to be updated since many major changes. Coming asap.\n\nThis is a Python wrapper for the MUSE/VLT pipeline recipes. This runs \nusing the esorex command lines (in parallel, using by default likwid, although \nother options are available). This wrapper is meant to reduce MUSE OBs automatically, \nafter dumping the fits files in a given folder.\n\nThis package includes an alignment module which may be useful, independently given \na set of images to align and assuming a reference image is provided. It can spit out \nan \"OFFSET_LIST\" MUSE compatible fits table including the flux scaling factors.\n\npymusepipe is also made for multi-pointing mosaics and multi-targets surveys\nas it will process targets automatically when provided a specific dictionary \nof which target and which pointings to consider.\n\n## 1 - Getting Started\nHere are some basic instructions to get you going with pymusepipe.\n\n### Prerequisites\npymusepipe uses Python 3. It is not compatible with python 2.7. If you are \nstill using python 2.7 I would recommend you to switch asap as it won't be \nsupported much longer anyway.\n\npymusepipe needs a number of standard modules/packages including:\n * **numpy**\n * **scipy** for the alignment module\n * **matplotlib** if you wish to make some plots (e.g. via the check_pipe module)\n * **astropy** as a general provider of many useful functions and structures, most importantly the astropy Tables.\n\nThe convolution package of pymusepipe (allowing full convolution of a datacube) uses\n**pypher** which is thus a requirement for that module to work.\n\nThis also includes a heavy-based usage of **mpdaf**, which is a super useful package\ndeveloped by the MUSE GTO-CRAL Team, to process and analyse datacubes, and more specifically\nMUSE cubes, images and spectra.\n\n### Installing\n\nYou can install this package via Pypi via a simple:\n```buildoutcfg\npip install pymusepipe\n```\n\nYou can obviously also install it by cloning it from github, or \ndownloading the source (from github) and do something like:\n```\npython setup.py develop\n```\nI advise to use the \"develop\" option as it actually does not copy \nthe files in your system but just create a link. \nIn that way you can easily update the source software without \nreinstalling it. The link will directly use the source which has been udpated.\n\nThe other option is to use the standard \"install\" option:\n```\npython setup.py install\n```\n\n## 2 - Basic Usage - Dealing with OBs, individually\nThe pymusepipe wrapper is meant to provide the user with a simple way to \nrun the MUSE pipeline.\n\nOnly 3 steps are needed: preparing the data (download), preparing the \nconfiguration files (templates are provided) which should take a couple of minutes, \nand using the code (a few lines at most). The package has many other hidden functionalities\nwhich I'll document as soon as I can.\n\nI recommend to use ```Ipython``` as an environment, possibly via \na ```screen``` which would allow you to disconnect from the server that \nactually runs the commands. Jupyter notebooks may also be very handy \nalthough I would not necessarily recommend them for this specific purpose as \nthe running of the pipeline may take hours.\n\n### Step 1 = Preparing your data\nThe first thing to do is to prepare the folder structure to welcome your \nMUSE datasets. This is actually rather simple.\n\nImagine you have\n * a target or field named \"NGC1000\" (just for the sake of using an example).\n * several pointings (each pointing can represent a set of OBs which have \n roughly the same coordinates, best is that they have a very significant \n overlap): 1, 2, and 3. Note that this 'pointing' structure may not reflect your\n own observational strategy. There are cases where one OB includes different pointings.\n What is important here is that each 'pointing' is either a full single OB, or several\n OBs with the same pointing.\n\nThen under your data root folder <my_data_folder> create the following folder structure:\n```\n<my_data_folder>/NGC1000\n ./P01\n ./Raw\n ./P02\n ./Raw\n ./P03\n ./Raw\n```\nEach pointing/OB has a \"Raw\" folder.\n\nThe next step is to download your MUSE data from the ESO web site, \nand put all the raw files (fits/fits.gz) into each individual \"Raw\" folder, \nassociated with the right pointing.\n\n### Step 2 = Preparing your configuration files\npymusepipe only needs 2 configurations ASCII files: 1 for the calibration \n(calib_tables.dic) and one for the root folders (rc.dic). \n**Examples** of such files are provided in the \"config_templates\" \nfolder of the pymusepipe package.\n\n### Step 3 = Running the pipeline\nThe pipeline is meant to be run automatically from a given python \nstructure. This should thus take the user only a few lines of codes, \nincluding one to import pymusepipe, a couple to define the names of \nthe configuration files, one to initialise the musepipe python structure, \nand one to launch the recipes. Here is an example of how this should look:\n\n```buildoutcfg\n# Import the modules\nimport pymusepipe as pmp\nfrom pymusepipe import musepipe\n\n# define the names of the configuration files\nrcfile = \"/my_data/MUSE/Config/rc.dic\"\ncalfile = \"my_data_MUSE/Config/calib_tables.dic\"\n\n# Initialisation of the python - MusePipe Class - structure\nmypipe = musepipe.MusePipe(targetname=\"NGCXXXX\", pointing=1, rc_filename=rcfile,\n cal_filename=calfile, log_filename=\"NGCXXXX_version01.log\",\n fakemode=False, overwrite_astropy_table=True, \n filter_list=\"white,Cousins_R\",\n filter_for_alignment=\"Cousins_R\")\n\n# Launching the pipeline\nmypipe.run_recipes()\n```\nSome explanation may be needed to understand what is happening:\n * \"targetname\": is just the name of the target, used to decided where the data will be\n * \"pointing\": will be used as \"P01\" for pointing=1, etc.\n * \"logfile\": name of the logging file. Note that this logfile is actually \n a shell-like file which can be used to re-run the pipeline one step at a time. Note that\n there will be also 2 more files created using that name <logfile_name>, namely:\n a file named <logfile_name>.out and <logfile_name>.err which will\n contain the full output of the commands, and the error messages (stdout, stderr, resp.).\n * \"fakemode\": you can set this to True if you just wish to initialise \n things without actually running any recipes. The pipeline will only set \n things up but if you run any recipes will only \"fake\" them (not launch any \n esorex command, only spitting the code out)\n * \"filter_list\": list of filter names to use to reconstruct images \n when building up cubes. This should be part of the filter_list fits \n table provided (see calib_tables config file).\n * \"filter_for_alignment\": specific filter name used for alignment between exposures.\n\nOther options can be useful:\n * \"musemode\": this is by default \"WFM_NOAO_N\" which is the most often \n used MUSE mode. This will filter out exposures not compatible with the \n given mode. So please beware.\n * \"reset_log\": will reset the log file. By default it is False, hence \n new runs will be appended.\n * \"overwrite_astropy_table\": by default this is False. If True, \n new runs will rewrite the Astropy output tables.\n * \"time_astrometry\": by default it is False, meaning the pipeline \n will try to detect a GEOMETRY and ASTROMETRY Files delivered with the Raw\n files by ESO. If set to True, it will use the time dependent astro/geo files\n provided by the GTO Team but you would need to make these available on your system.\n Hence I would recommend to keep the default (False).\n\n## 3- Advanced Usage - Targets and Mosaics\n\n## 4- Alignment module\nDetails to come.\n\n## 5- Further details\nHere we provide a bit more details about a few issues regarding \nthe configuration files or set up parameters.\n\n### Configuration files\n\n#### rc configuration file\nIt contains 3 lines, with: *musecalib*, *musecalib_time* and *root*.\n\n * *root* provides the root folder for your data. For Target NGCXXXX, \n and Pointing 1, the Raw data will be looked for \n in *root*/TargetXXXX/P01/Raw (see *Preparing your data* above).\n * musecalib should contain the standard MUSE calibration files. \n These are usually distributed in the MUSE kit within a \n \"muse-calib-x.x.x/cal\" folder.\n * musecalib_time: time dependent geometry and astrometry files \n (the correspondence between observing run dates and specific \n files are given in the dic_geo_astrowcs_table in musepipe.py).\n\n#### calib_tables configuration file\nIt contains a series of given fits files which will be used by the pipeline. Most names are self-explantory. That includes:\n * *geo_table* and *astro_table*: only used if you don't rely on the default time dependent geometry files (see rc file).\n * *badpix_table*, *vignetting_mask*, *std_flux_table*, *extinct_table*, *line_catalog* all usually provided with the MUSE pipeline.\n * *filter_list* : name of the fits filter list. This is used in case you wish to provide your own. Note that it needs to follow the MUSE standard for such a table.\n\n### Recipes\nMost MUSE pipeline recipes are run while run_all_recipes is launched. This can be changed in the prep_recipes_pipe.py or just scripted. Actually run_all_recipes() is just a function which launches (assuming 'mypipe' is your MusePipe structure, see above):\n```\n mypipe.run_bias()\n mypipe.run_flat()\n mypipe.run_wave()\n mypipe.run_lsf()\n mypipe.run_twilight(illum=illum)\n mypipe.run_scibasic_all(illum=illum)\n mypipe.run_standard()\n mypipe.run_sky(fraction=fraction)\n mypipe.run_prep_align()\n mypipe.run_align_bypointing()\n mypipe.run_align_bygroup()\n mypipe.run_scipost()\n mypipe.run_scipost(expotype=\"SKY\", offset_list=False, skymethod='none')\n mypipe.run_combine_pointing()\n```\nwhere \"illum\" is a boolean (default is True), fraction is 0.8.\n\nFeel free to launch these steps one by one (the order is important as in any data reduction process).\n\n### Structure of the output\n\n#### Folders\nThe structure of the output is driven by a set of folder names described in init_musepipe.py in a few dictionaries (dic_input_folders, dic_folders, dic_folders_target). You can in principle change the names of the folders themselves, but I would advise against that.\n\nThe pipeline will create the folder structure automatically, checking whether the folders exist or not.\n\n#### Log files\n2 basic log files are spitted out: one is the Esorex output which will \nbe stored in the \"Esorex_log\" folder. The other one will be in the \"Log\" \nfolder with the name provided at start: that one is like a shell script \nwhich can be used to rerun things directly via the command line.\nIn the \"Log\" folder, there will also be, for each log file, a file \".out\" and\none with \".err\" extensions, respectively including all the stdout and stderr\nmessages. This may be useful to trace details in the data reduction and problems.\n\n#### Astropy Tables\nEach recipe will trigger the creation of a astropy Table. \nThese are stored under \"Astro_Tables\". You can use these to monitor \nwhich files have been processed or used.\n\n#### Python structure\nMost of the information you may need is actually stored in the \npython \"MusePipe\" class structure. More details to come.\n\n## Authors\n* **Eric Emsellem** [2017-2020], at ESO and CRAL\n\n## License\n\nThis project is licensed under the MIT License - see the \n[LICENSE](LICENSE) file for details\n\n## Acknowledgments\nI would like to thank people who have initially sent me \ntheir code-samples, including Bernd Husemann, Dimitri Gadotti, \nLodovico Coccato, Mark den Brok. I would also like to specifically \nand warmly thank Rebecca McElroy who supported me with the MUSE\ndata reduction at the early stages of the development of this package,\nand Francesco Santoro who has significantly contributed \nin the testing, debugging of the code (version 1), and proposed dedicated python lines \nto be integrated in pymusepipe (e.g., alignment module).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "python module to reduce MUSE Raw data and combine them",
"version": "2.28.0",
"project_urls": {
"Download": "https://github.com/emsellem/pymusepipe/archive/v2.9.6.beta.tar.gz",
"Homepage": "https://github.com/emsellem/pymusepipe"
},
"split_keywords": [
"muse",
"datareduction"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "64f09ba2f435596929c4799cbd521faaa298f40951b489face4267058412b64b",
"md5": "88140ecdf9f44935357aa06f2dbbdb6d",
"sha256": "59f1444ff73b31ca6298a62436126015ac0a9b6123c38d50c67563c3ffda7fb4"
},
"downloads": -1,
"filename": "pymusepipe-2.28.0.tar.gz",
"has_sig": false,
"md5_digest": "88140ecdf9f44935357aa06f2dbbdb6d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 7552716,
"upload_time": "2024-03-13T14:33:20",
"upload_time_iso_8601": "2024-03-13T14:33:20.421264Z",
"url": "https://files.pythonhosted.org/packages/64/f0/9ba2f435596929c4799cbd521faaa298f40951b489face4267058412b64b/pymusepipe-2.28.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-13 14:33:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "emsellem",
"github_project": "pymusepipe",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "astropy",
"specs": [
[
">=",
"4.0"
]
]
},
{
"name": "mpdaf",
"specs": [
[
">=",
"3.5"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.8"
]
]
},
{
"name": "scipy",
"specs": [
[
">=",
"1.4.1"
]
]
},
{
"name": "matplotlib",
"specs": [
[
">=",
"3.0.0"
]
]
},
{
"name": "sphinxcontrib.email",
"specs": []
},
{
"name": "reproject",
"specs": [
[
">=",
"0.8"
]
]
},
{
"name": "scikit-image",
"specs": [
[
">=",
"0.19.3"
]
]
},
{
"name": "spacepylot",
"specs": [
[
">=",
"0.0.post1.dev20+g857829b.d20221118"
]
]
},
{
"name": "pymusepipe",
"specs": [
[
">=",
"2.26.9"
]
]
},
{
"name": "photutils",
"specs": [
[
">=",
"1.2.0"
]
]
},
{
"name": "pypher",
"specs": [
[
">=",
"0.6.4"
]
]
},
{
"name": "sphinx",
"specs": [
[
">=",
"5.3.0"
]
]
},
{
"name": "sphinx_rtd_theme",
"specs": [
[
">=",
"1.2.0"
]
]
},
{
"name": "setuptools",
"specs": [
[
">=",
"67.7.2"
]
]
}
],
"lcname": "pymusepipe"
}
JSON
