| Name | pyread-swift JSON |
| Version |
1.0.3
JSON |
| download |
| home_page | |
| Summary | Package to read SWIFT simulation snapshots in MPI. |
| upload_time | 2023-06-15 19:20:55 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | BSD 3-Clause License Copyright (c) 2022, LSST Dark Energy Science Collaboration (DESC) All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| keywords |
swiftsim
simulations
mpi
numerical
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
## An MPI read routine for Swift simulation snapshots
``pyread_swift`` is an MPI read routine for [``swiftsim``](https://github.com/SWIFTSIM/swiftsim) snapshots, very similar in style to John Helly's [``read_eagle``](https://gitlab.cosma.dur.ac.uk/jch/Read_Eagle) code to read EAGLE snapshots.
The package can read ``swiftsim`` snapshots both in "collective" (i.e., multiple MPI ranks read from a single file simultaneously) and "distributed" (i.e., each MPI reads an individual snapshot file part in isolation) modes.
## Installation
### Requirements
* `OpenMPI` or other MPI library
* `python>=3.8`
Recommended modules when working on COSMA7:
```bash
module load gnu_comp/11.1.0 openmpi/4.1.4 parallel_hdf5/1.12.0 python/3.9.1-C7
```
Given the need for a parallel HDF5 installation, it is recommended you install ``pyread_swift`` within a virtual/conda environment. However you can ofcourse also install directly into your base Python environment if you prefer.
First make sure your `pip` is up-to-date:
```bash
python3 -m pip install --upgrade pip
```
### Method 1) Installation from PyPi
The easiest method is to install from [``PyPI``](https://pypi.org/project/pyread-swift/)
```bash
python3 -m pip install pyread-swift
```
### Method 2) Installation from source
Or, you can install directly from source.
First clone the repo, then you can install the `pyread_swift` package by typing the following in
the root git directory:
```bash
git clone https://github.com/stuartmcalpine/pyread_swift.git
cd pyread_swift
python3 -m pip install .
```
which will install `pyread_swift` and any dependencies.
### MPI installation for collective reading
If you are using `pyread_swift` to load large snapshots over MPI collectively
(i.e., multiple cores read in parallel from the same file), a bit of additional
setup is required.
Make sure you have `hdf5` installed with **parallel** compatibility ([see here for details](https://docs.h5py.org/en/stable/mpi.html)).
Then, uninstall any versions of `h5py` and reinstall from source:
```bash
python3 -m pip uninstall h5py
MPICC=mpicc CC=mpicc HDF5_MPI="ON" python3 -m pip install --no-binary=h5py h5py
```
If `pip` struggles to find your `HDF5` libraries automatically, e.g., `error: libhdf5.so: cannot open shared object file: No such file or directory`. You may have to specify the path to the HDF5 installation manually, i.e., `HDF5_DIR=/path/to/hdf5/lib` (see [here](https://docs.h5py.org/en/stable/build.html#building-against-parallel-hdf5) for more details).
For our COSMA7 setup, that would be:
`HDF5_DIR="/cosma/local/parallel-hdf5//gnu_11.1.0_ompi_4.1.4/1.12.0/"`
## Usage
``pyread_swift`` is build around a primary read wrapper, called ``SwiftSnapshot``. The snapshot particles are loaded into, stored, and manipulated by this object.
Reading follows these four steps (see also the examples below):
* Initialize a ``SwiftSnapshot`` object pointing to the location of the HDF5 file.
* Select the spatial region you want to extract the particles from using the ``select_region()`` routine.
* Split the selection over the MPI ranks using the ``split_selection()`` routine.
* Read a selected property of the particles using the ``read_dataset()`` routine.
### Input parameters to SwiftSnapshot
| Input | Description | Default option |
| ----- | ----------- | --------- |
| fname | Full path to HDF5 snapshot file. If the snapshot is split over multiple files, this can just be one of the file parts | - |
| comm= | MPI4PY communicator (if reading in MPI) | None |
| verbose= | True for more a more verbose output | False |
| mpi_read_format= | How to read the snapshot in MPI mode ("collective" or "distributed") <br><br>"collective": Do a collective read of each file, i.e., all ranks read a single file at one. Recommended for single, or few large snapshot file(s). Requires parallel-hdf5 to be installed. <br><br>"distributed": Each rank reads its own file part. Recommended for multiple smaller files. | "collective" |
| max_concur_io= | When reading in MPI, how many HDF5 files can be open at once | 64 |
### Example usage (No MPI case)
```python
from pyread_swift import SwiftSnapshot
# Set up pyread_swift object pointing at HDF5 snapshot file (or a file part).
snapshot = "/path/to/snap/part.0.hdf5"
swift = SwiftSnapshot(snapshot)
# Select region to load from.
parttype = 1 # Dark matter
region = [0,100,0,100,0,100] # [xlo,xhi,ylo,yhi,zlo,zhi]
swift.select_region(parttype, *region)
# Divide selection between ranks (needs to be invoked even for non-mpi case).
swift.split_selection()
# Read data.
ids = swift.read_dataset(parttype, "ParticleIDs")
```
### Example usage (MPI case)
```python
from mpi4py import MPI
from pyread_swift import SwiftSnapshot
# MPI communicator.
comm = MPI.COMM_WORLD
# Set up read_swift object pointing at HDF5 snapshot file (or a file part).
snapshot = "/path/to/snap/part.0.hdf5"
swift = SwiftSnapshot(snapshot, comm=comm)
# Select region to load from.
parttype = 1 # Dark matter
region = [0,100,0,100,0,100] # [xlo,xhi,ylo,yhi,zlo,zhi]
swift.select_region(parttype, *region)
# Divide selection between ranks.
swift.split_selection()
# Read data.
ids = swift.read_dataset(parttype, "ParticleIDs")
```
Raw data
{
"_id": null,
"home_page": "",
"name": "pyread-swift",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "swiftsim,simulations,mpi,numerical",
"author": "",
"author_email": "Stuart McAlpine <stuart.mcalpine@fysik.su.se>",
"download_url": "https://files.pythonhosted.org/packages/94/6c/2801fe8c8d90d8f0558aee826d3a1fdd4112db94ee986cbb46861dec3c1d/pyread_swift-1.0.3.tar.gz",
"platform": null,
"description": "## An MPI read routine for Swift simulation snapshots\n\n``pyread_swift`` is an MPI read routine for [``swiftsim``](https://github.com/SWIFTSIM/swiftsim) snapshots, very similar in style to John Helly's [``read_eagle``](https://gitlab.cosma.dur.ac.uk/jch/Read_Eagle) code to read EAGLE snapshots.\n\nThe package can read ``swiftsim`` snapshots both in \"collective\" (i.e., multiple MPI ranks read from a single file simultaneously) and \"distributed\" (i.e., each MPI reads an individual snapshot file part in isolation) modes. \n\n## Installation\n\n### Requirements\n\n* `OpenMPI` or other MPI library\n* `python>=3.8`\n\nRecommended modules when working on COSMA7:\n\n```bash\nmodule load gnu_comp/11.1.0 openmpi/4.1.4 parallel_hdf5/1.12.0 python/3.9.1-C7\n```\n\nGiven the need for a parallel HDF5 installation, it is recommended you install ``pyread_swift`` within a virtual/conda environment. However you can ofcourse also install directly into your base Python environment if you prefer.\n\nFirst make sure your `pip` is up-to-date:\n\n```bash\npython3 -m pip install --upgrade pip\n```\n\n### Method 1) Installation from PyPi\n\nThe easiest method is to install from [``PyPI``](https://pypi.org/project/pyread-swift/)\n\n```bash\npython3 -m pip install pyread-swift\n```\n\n### Method 2) Installation from source\n\nOr, you can install directly from source.\n\nFirst clone the repo, then you can install the `pyread_swift` package by typing the following in\nthe root git directory: \n\n```bash\ngit clone https://github.com/stuartmcalpine/pyread_swift.git\ncd pyread_swift\npython3 -m pip install .\n```\n\nwhich will install `pyread_swift` and any dependencies.\n\n### MPI installation for collective reading\n\nIf you are using `pyread_swift` to load large snapshots over MPI collectively\n(i.e., multiple cores read in parallel from the same file), a bit of additional\nsetup is required.\n\nMake sure you have `hdf5` installed with **parallel** compatibility ([see here for details](https://docs.h5py.org/en/stable/mpi.html)).\n\nThen, uninstall any versions of `h5py` and reinstall from source:\n\n```bash\npython3 -m pip uninstall h5py\nMPICC=mpicc CC=mpicc HDF5_MPI=\"ON\" python3 -m pip install --no-binary=h5py h5py\n```\n\nIf `pip` struggles to find your `HDF5` libraries automatically, e.g., `error: libhdf5.so: cannot open shared object file: No such file or directory`. You may have to specify the path to the HDF5 installation manually, i.e., `HDF5_DIR=/path/to/hdf5/lib` (see [here](https://docs.h5py.org/en/stable/build.html#building-against-parallel-hdf5) for more details).\n\nFor our COSMA7 setup, that would be:\n\n`HDF5_DIR=\"/cosma/local/parallel-hdf5//gnu_11.1.0_ompi_4.1.4/1.12.0/\"`\n\n## Usage\n\n``pyread_swift`` is build around a primary read wrapper, called ``SwiftSnapshot``. The snapshot particles are loaded into, stored, and manipulated by this object.\n\nReading follows these four steps (see also the examples below):\n\n* Initialize a ``SwiftSnapshot`` object pointing to the location of the HDF5 file.\n\n* Select the spatial region you want to extract the particles from using the ``select_region()`` routine.\n\n* Split the selection over the MPI ranks using the ``split_selection()`` routine.\n\n* Read a selected property of the particles using the ``read_dataset()`` routine.\n\n### Input parameters to SwiftSnapshot\n\n| Input | Description | Default option |\n| ----- | ----------- | --------- |\n| fname | Full path to HDF5 snapshot file. If the snapshot is split over multiple files, this can just be one of the file parts | - |\n| comm= | MPI4PY communicator (if reading in MPI) | None |\n| verbose= | True for more a more verbose output | False |\n| mpi_read_format= | How to read the snapshot in MPI mode (\"collective\" or \"distributed\") <br><br>\"collective\": Do a collective read of each file, i.e., all ranks read a single file at one. Recommended for single, or few large snapshot file(s). Requires parallel-hdf5 to be installed. <br><br>\"distributed\": Each rank reads its own file part. Recommended for multiple smaller files. | \"collective\" |\n| max_concur_io= | When reading in MPI, how many HDF5 files can be open at once | 64 |\n\n### Example usage (No MPI case)\n\n```python\nfrom pyread_swift import SwiftSnapshot\n\n# Set up pyread_swift object pointing at HDF5 snapshot file (or a file part). \nsnapshot = \"/path/to/snap/part.0.hdf5\"\nswift = SwiftSnapshot(snapshot)\n\n# Select region to load from.\nparttype = 1 # Dark matter\nregion = [0,100,0,100,0,100] # [xlo,xhi,ylo,yhi,zlo,zhi]\nswift.select_region(parttype, *region)\n\n# Divide selection between ranks (needs to be invoked even for non-mpi case).\nswift.split_selection()\n\n# Read data.\nids = swift.read_dataset(parttype, \"ParticleIDs\")\n```\n\n### Example usage (MPI case)\n\n```python\nfrom mpi4py import MPI\nfrom pyread_swift import SwiftSnapshot\n\n# MPI communicator.\ncomm = MPI.COMM_WORLD\n\n# Set up read_swift object pointing at HDF5 snapshot file (or a file part). \nsnapshot = \"/path/to/snap/part.0.hdf5\"\nswift = SwiftSnapshot(snapshot, comm=comm)\n\n# Select region to load from.\nparttype = 1 # Dark matter\nregion = [0,100,0,100,0,100] # [xlo,xhi,ylo,yhi,zlo,zhi]\nswift.select_region(parttype, *region)\n\n# Divide selection between ranks.\nswift.split_selection()\n\n# Read data.\nids = swift.read_dataset(parttype, \"ParticleIDs\")\n```\n\n\n",
"bugtrack_url": null,
"license": "BSD 3-Clause License Copyright (c) 2022, LSST Dark Energy Science Collaboration (DESC) All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ",
"summary": "Package to read SWIFT simulation snapshots in MPI.",
"version": "1.0.3",
"project_urls": null,
"split_keywords": [
"swiftsim",
"simulations",
"mpi",
"numerical"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2db8cdf6e5b552b3f76d7cb15dc4a024eef218ae5684f4f363eb5e8705e18208",
"md5": "66b4307f9a9eabcc9a7428c5a833b008",
"sha256": "8d82eb1d48a0fef97c44326c51f8a0e86f8d3ac77b69cf60dc59808e0076b547"
},
"downloads": -1,
"filename": "pyread_swift-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "66b4307f9a9eabcc9a7428c5a833b008",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 16702,
"upload_time": "2023-06-15T19:20:53",
"upload_time_iso_8601": "2023-06-15T19:20:53.447668Z",
"url": "https://files.pythonhosted.org/packages/2d/b8/cdf6e5b552b3f76d7cb15dc4a024eef218ae5684f4f363eb5e8705e18208/pyread_swift-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "946c2801fe8c8d90d8f0558aee826d3a1fdd4112db94ee986cbb46861dec3c1d",
"md5": "1062fef0c95ad276e58fac5fd4b39221",
"sha256": "8073d83f83abe41cafba79e84a145d1ccf7d4bb5b0aa6186d117d80c7fa3e4cb"
},
"downloads": -1,
"filename": "pyread_swift-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "1062fef0c95ad276e58fac5fd4b39221",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 17020,
"upload_time": "2023-06-15T19:20:55",
"upload_time_iso_8601": "2023-06-15T19:20:55.847933Z",
"url": "https://files.pythonhosted.org/packages/94/6c/2801fe8c8d90d8f0558aee826d3a1fdd4112db94ee986cbb46861dec3c1d/pyread_swift-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-15 19:20:55",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "pyread-swift"
}
