[](http://www.gnu.org/licenses/gpl-3.0)
[](https://doi.org/10.1038/s41593-021-00947-w)
# DeepMReye: magnetic resonance-based eye tracking using deep neural networks
This [Jupyter Notebook](./notebooks/deepmreye_example_usage.ipynb) provides a step-by-step walkthrough of the code. It includes eyeball coregistration, voxel extraction, model training and test as well as basic performance measures. Alternatively, here is a [Colab Notebook](https://colab.research.google.com/drive/1kYVyierbKdNZ3RY4_pbACtdWEw7PKQuz?usp=sharing).
This [Data Repository](https://osf.io/mrhk9/) includes exemplary data for model training and test, source data of all paper figures as well as pre-trained model weights.
Moreover, here are additional [User Recommendations](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/kKdOXmLqe) as well as a [Frequently-Asked-Questions (FAQ)](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/sargIAQ6t) page. If you have other questions, please reach out to us.
## Installation
### Option 1: Pip install
#### Pip installation
Install DeepMReye with a CPU/GPU version of [TensorFlow](https://www.tensorflow.org/install/) using the following command.
```
pip install deepmreye
```
#### Anaconda / Miniconda installation
To encapsulate DeepMReye in a virtual environment install with the following commands:
```
conda create --name deepmreye python=3.9
conda activate deepmreye
pip install deepmreye
```
For GPU support, follow tensorflow [install](https://www.tensorflow.org/install/pip) instruction, e.g.:
```
conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
```
If installation of [ANTsPy](https://github.com/ANTsX/ANTsPy) fails try to manually install it via:
```
git clone https://github.com/ANTsX/ANTsPy
cd ANTsPy
pip install CMake
python3 setup.py install
```
### Option 2: Colab
We provide a [Colab Notebook](https://colab.research.google.com/drive/1kYVyierbKdNZ3RY4_pbACtdWEw7PKQuz?usp=sharing) showcasing model training and evaluation on a GPU provided by Google Colab. To use your own data, preprocess your data locally and upload only the extracted eyeball voxels. This saves space and avoids data privacy issues. See the [Jupyter Notebook](./notebooks/deepmreye_example_usage.ipynb) for the preprocessing and eyeball-extraction code.
[](https://colab.research.google.com/drive/1kYVyierbKdNZ3RY4_pbACtdWEw7PKQuz?usp=sharing)
### Option 3: Docker
Pull the image from docker hub.
```bash
docker pull deepmreye/deepmreye
```
Use deepMReye in a docker container via jupyterlab:
```bash
mkdir -p $PWD/notebooks
docker run -it --rm \
--publish 8888:8888 \
--volume $PWD/notebooks:/home/neuro/notebooks \
deepmreye/deepmreye:latest \
jupyter-lab --no-browser --ip 0.0.0.0
```
### Option 4: Streamlit (Browser version)
If you would just like to try it out and decode gaze coordinates in your data using a pretrained model, the easiest way is using our streamlit app.
Running the following commands will open a browser window that allows you to upload your data (.nii or .nii.gz) and then download the corresponding gaze coordinates shortly after.
Please read the [User Recommendations](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/kKdOXmLqe) before using the pretrained models.
```bash
git clone https://github.com/DeepMReye/DeepMReye.git
cd DeepMReye
pip install .
pip install streamlit
streamlit run streamlit/streamlit.py
```
### Data formats
The <u>**fMRI data**</u> should be organized in 4D NIFTI files (.nii), containing the realigned 3D images acquired over time. The pipeline then extracts the eyeball voxels automatically and saves them as Python Pickle files, which serve as model input. For model training, you additionally need <u>**training labels**</u>, a numpy array containing 10 gaze coordinates per functional volume. These gaze coordinates can either be camera-based eye-tracking labels or the coordinates of a fixation target, and many file formats can be easily read (e.g. .npy, .npz, .mat, .csv etc.).
Please see our [FAQ](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/sargIAQ6t) page for more details on data formats and preprocessing.
## Hardware requirements
The GPU version of DeepMReye requires a NVIDIA GPU.
## Software requirements
The following python dependencies are being automatically installed when installing DeepMReye (specified in setup.cfg):
```
tensorflow-gpu (2.2.0)
numpy (1.19.1)
pandas (1.0.5)
matplotlib (3.2.2)
scipy (1.5.0)
ipython (7.13.0)
plotly (4.14.3)
```
Version in parentheses indicate the ones used for testing the framework. Its extensively tested on Linux 16.04 but should run on all OS (Windows, Mac, Linux) supporting a Python version >3.8 and pip. It is recommended to install the framework and dependencies in a virtual environment (e.g. conda).
## BIDS app
If you would like to run a pretrained version of DeepMReye on datasets that follow the Brain Imaging Data Structure (BIDS) format, check out [bidsMReye](https://pypi.org/project/bidsmreye/): a wrapper for DeepMReye for BIDS datasets (incl. [fMRIprep](https://fmriprep.org/en/stable/) outputs) developed by [Remi Gau](https://remi-gau.github.io/). Please read the paper and [User Recommendations](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/kKdOXmLqe) before you use it.
## Correspondence
If you have questions, comments or inquiries, please check out the online [User documention](https://deepmreye.slite.com/api/s/channel/MUgmvViEbaATSrqt3susLZ/DeepMReye%3A%20Documentation) and reach out to us: markus.frey[at]ntnu.no & matthias.nau[at]ntnu.no
Raw data
{
"_id": null,
"home_page": "https://github.com/DeepMReye/DeepMReye",
"name": "deepmreye",
"maintainer": "Markus Frey",
"docs_url": null,
"requires_python": ">=3.8.0",
"maintainer_email": "markus.frey1@gmail.com",
"keywords": "",
"author": "Markus Frey",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/73/a6/bab7239943dd72fdf232b404b10dc913728c163708f0392c847d68def28b/deepmreye-0.2.1.tar.gz",
"platform": "OS Independent",
"description": "[](http://www.gnu.org/licenses/gpl-3.0)\n\n\n[](https://doi.org/10.1038/s41593-021-00947-w)\n\n\n\n# DeepMReye: magnetic resonance-based eye tracking using deep neural networks\nThis [Jupyter Notebook](./notebooks/deepmreye_example_usage.ipynb) provides a step-by-step walkthrough of the code. It includes eyeball coregistration, voxel extraction, model training and test as well as basic performance measures. Alternatively, here is a [Colab Notebook](https://colab.research.google.com/drive/1kYVyierbKdNZ3RY4_pbACtdWEw7PKQuz?usp=sharing).\n\nThis [Data Repository](https://osf.io/mrhk9/) includes exemplary data for model training and test, source data of all paper figures as well as pre-trained model weights.\n\nMoreover, here are additional [User Recommendations](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/kKdOXmLqe) as well as a [Frequently-Asked-Questions (FAQ)](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/sargIAQ6t) page. If you have other questions, please reach out to us.\n\n\n\n## Installation\n\n### Option 1: Pip install\n\n#### Pip installation\nInstall DeepMReye with a CPU/GPU version of [TensorFlow](https://www.tensorflow.org/install/) using the following command.\n```\npip install deepmreye\n```\n\n#### Anaconda / Miniconda installation\n\nTo encapsulate DeepMReye in a virtual environment install with the following commands:\n```\nconda create --name deepmreye python=3.9\nconda activate deepmreye\npip install deepmreye\n```\nFor GPU support, follow tensorflow [install](https://www.tensorflow.org/install/pip) instruction, e.g.:\n```\nconda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0\n```\nIf installation of [ANTsPy](https://github.com/ANTsX/ANTsPy) fails try to manually install it via:\n```\ngit clone https://github.com/ANTsX/ANTsPy\ncd ANTsPy\npip install CMake\npython3 setup.py install\n```\n\n### Option 2: Colab\n\nWe provide a [Colab Notebook](https://colab.research.google.com/drive/1kYVyierbKdNZ3RY4_pbACtdWEw7PKQuz?usp=sharing) showcasing model training and evaluation on a GPU provided by Google Colab. To use your own data, preprocess your data locally and upload only the extracted eyeball voxels. This saves space and avoids data privacy issues. See the [Jupyter Notebook](./notebooks/deepmreye_example_usage.ipynb) for the preprocessing and eyeball-extraction code.\n\n[](https://colab.research.google.com/drive/1kYVyierbKdNZ3RY4_pbACtdWEw7PKQuz?usp=sharing)\n\n\n\n### Option 3: Docker\n\nPull the image from docker hub.\n\n```bash\ndocker pull deepmreye/deepmreye\n```\n\nUse deepMReye in a docker container via jupyterlab:\n\n```bash\nmkdir -p $PWD/notebooks\ndocker run -it --rm \\\n --publish 8888:8888 \\\n --volume $PWD/notebooks:/home/neuro/notebooks \\\n deepmreye/deepmreye:latest \\\n jupyter-lab --no-browser --ip 0.0.0.0\n```\n\n### Option 4: Streamlit (Browser version)\n\nIf you would just like to try it out and decode gaze coordinates in your data using a pretrained model, the easiest way is using our streamlit app. \n\nRunning the following commands will open a browser window that allows you to upload your data (.nii or .nii.gz) and then download the corresponding gaze coordinates shortly after. \nPlease read the [User Recommendations](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/kKdOXmLqe) before using the pretrained models.\n\n```bash\ngit clone https://github.com/DeepMReye/DeepMReye.git\ncd DeepMReye\npip install .\npip install streamlit\nstreamlit run streamlit/streamlit.py\n```\n\n### Data formats\nThe <u>**fMRI data**</u> should be organized in 4D NIFTI files (.nii), containing the realigned 3D images acquired over time. The pipeline then extracts the eyeball voxels automatically and saves them as Python Pickle files, which serve as model input. For model training, you additionally need <u>**training labels**</u>, a numpy array containing 10 gaze coordinates per functional volume. These gaze coordinates can either be camera-based eye-tracking labels or the coordinates of a fixation target, and many file formats can be easily read (e.g. .npy, .npz, .mat, .csv etc.).\n\nPlease see our [FAQ](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/sargIAQ6t) page for more details on data formats and preprocessing.\n\n## Hardware requirements\n\nThe GPU version of DeepMReye requires a NVIDIA GPU.\n\n## Software requirements\nThe following python dependencies are being automatically installed when installing DeepMReye (specified in setup.cfg):\n```\ntensorflow-gpu (2.2.0)\nnumpy (1.19.1)\npandas (1.0.5)\nmatplotlib (3.2.2)\nscipy (1.5.0)\nipython (7.13.0)\nplotly (4.14.3)\n```\nVersion in parentheses indicate the ones used for testing the framework. Its extensively tested on Linux 16.04 but should run on all OS (Windows, Mac, Linux) supporting a Python version >3.8 and pip. It is recommended to install the framework and dependencies in a virtual environment (e.g. conda). \n\n## BIDS app\nIf you would like to run a pretrained version of DeepMReye on datasets that follow the Brain Imaging Data Structure (BIDS) format, check out [bidsMReye](https://pypi.org/project/bidsmreye/): a wrapper for DeepMReye for BIDS datasets (incl. [fMRIprep](https://fmriprep.org/en/stable/) outputs) developed by [Remi Gau](https://remi-gau.github.io/). Please read the paper and [User Recommendations](https://deepmreye.slite.com/p/channel/MUgmvViEbaATSrqt3susLZ/notes/kKdOXmLqe) before you use it.\n\n## Correspondence\nIf you have questions, comments or inquiries, please check out the online [User documention](https://deepmreye.slite.com/api/s/channel/MUgmvViEbaATSrqt3susLZ/DeepMReye%3A%20Documentation) and reach out to us: markus.frey[at]ntnu.no & matthias.nau[at]ntnu.no\n",
"bugtrack_url": null,
"license": "LGPL-3.0",
"summary": "MR-based eye tracker without eye tracking",
"version": "0.2.1",
"project_urls": {
"Download": "https://github.com/DeepMReye/DeepMReye",
"Homepage": "https://github.com/DeepMReye/DeepMReye"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3256c2d01a650a8bfb457886a6dba37f590b91c322e97954dbf5b69bd227fa75",
"md5": "7e4b3389bcc2ed5acf1489c25869dd1a",
"sha256": "7e3dd47637856d13e37b7fa2344712ef84092bf2b7a33ce6022999f93432bb55"
},
"downloads": -1,
"filename": "deepmreye-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7e4b3389bcc2ed5acf1489c25869dd1a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0",
"size": 2870910,
"upload_time": "2023-06-16T19:24:30",
"upload_time_iso_8601": "2023-06-16T19:24:30.536685Z",
"url": "https://files.pythonhosted.org/packages/32/56/c2d01a650a8bfb457886a6dba37f590b91c322e97954dbf5b69bd227fa75/deepmreye-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "73a6bab7239943dd72fdf232b404b10dc913728c163708f0392c847d68def28b",
"md5": "a57419a6145036786d809878a7a9f029",
"sha256": "0799dcb703841610ee172fb439e87ad2977eda2d5d93b749b7bb2c4c4295d920"
},
"downloads": -1,
"filename": "deepmreye-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "a57419a6145036786d809878a7a9f029",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0",
"size": 2874879,
"upload_time": "2023-06-16T19:24:32",
"upload_time_iso_8601": "2023-06-16T19:24:32.639068Z",
"url": "https://files.pythonhosted.org/packages/73/a6/bab7239943dd72fdf232b404b10dc913728c163708f0392c847d68def28b/deepmreye-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-16 19:24:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DeepMReye",
"github_project": "DeepMReye",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"tox": true,
"lcname": "deepmreye"
}
JSON
