| Name | ivy-memory JSON |
| Version |
1.1.9
JSON |
| download |
| home_page | https://ivy-dl.org/memory |
| Summary | End-to-end memory modules for machine learning developers, written in Ivy. |
| upload_time | 2021-12-01 16:23:11 |
| maintainer | |
| docs_url | None |
| author | Ivy Team |
| requires_python | |
| license | Apache 2.0 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
.. image:: https://github.com/ivy-dl/memory/blob/master/docs/partial_source/logos/logo.png?raw=true
:width: 100%
**End-to-end memory modules for machine learning developers, written in Ivy.**
.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/logos/supported/frameworks.png?raw=true
:width: 100%
Contents
--------
* `Overview`_
* `Run Through`_
* `Interactive Demos`_
* `Get Involed`_
Overview
--------
.. _docs: https://ivy-dl.org/memory
**What is Ivy Memory?**
Ivy memory provides differentiable memory modules,
including learnt modules such as Neural Turing Machines (NTM),
but also parameter-free modules such as End-to-End Egospheric Spatial Memory (ESM).
Check out the docs_ for more info!
The library is built on top of the Ivy machine learning framework.
This means all memory modules simultaneously support:
Jax, Tensorflow, PyTorch, MXNet, and Numpy.
**Ivy Libraries**
There are a host of derived libraries written in Ivy, in the areas of mechanics, 3D vision, robotics, gym environments,
neural memory, pre-trained models + implementations, and builder tools with trainers, data loaders and more. Click on
the icons below to learn more!
.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/ivy_libraries.png?raw=true
:width: 100%
**Quick Start**
Ivy memory can be installed like so: ``pip install ivy-memory``
.. _demos: https://github.com/ivy-dl/memory/tree/master/ivy_memory_demos
.. _interactive: https://github.com/ivy-dl/memory/tree/master/ivy_memory_demos/interactive
To quickly see the different aspects of the library, we suggest you check out the demos_!
We suggest you start by running the script ``run_through.py``,
and read the "Run Through" section below which explains this script.
For more interactive demos, we suggest you run either
``learning_to_copy_with_ntm.py`` or ``mapping_a_room_with_esm.py`` in the interactive_ demos folder.
Run Through
-----------
We run through some of the different parts of the library via a simple ongoing example script.
The full script is available in the demos_ folder, as file ``run_through.py``.
**End-to-End Egospheric Spatial Memory**
First, we show how the Ivy End-to-End Egospheric Spatial Memory (ESM) class can be used inside a pure-Ivy model.
We first define the model as below.
.. code-block:: python
class IvyModelWithESM(ivy.Module):
def __init__(self, channels_in, channels_out):
self._channels_in = channels_in
self._esm = ivy_mem.ESM(omni_image_dims=(16, 32))
self._linear = ivy_mem.Linear(channels_in, channels_out)
ivy.Module.__init__(self, 'cpu')
def _forward(self, obs):
mem = self._esm(obs)
x = ivy.reshape(mem.mean, (-1, self._channels_in))
return self._linear(x)
Next, we instantiate this model, and verify that the returned tensors are of the expected shape.
.. code-block:: python
# create model
in_channels = 32
out_channels = 8
ivy.set_framework('torch')
model = IvyModelWithESM(in_channels, out_channels)
# input config
batch_size = 1
image_dims = [5, 5]
num_timesteps = 2
num_feature_channels = 3
# create image of pixel co-ordinates
uniform_pixel_coords =\
ivy_vision.create_uniform_pixel_coords_image(image_dims, [batch_size, num_timesteps])
# define camera measurement
depths = ivy.random_uniform(shape=[batch_size, num_timesteps] + image_dims + [1])
pixel_coords = ivy_vision.depth_to_pixel_coords(depths)
inv_calib_mats = ivy.random_uniform(shape=[batch_size, num_timesteps, 3, 3])
cam_coords = ivy_vision.pixel_to_cam_coords(pixel_coords, inv_calib_mats)[..., 0:3]
features = ivy.random_uniform(shape=[batch_size, num_timesteps] + image_dims + [num_feature_channels])
img_mean = ivy.concatenate((cam_coords, features), -1)
cam_rel_mat = ivy.identity(4, batch_shape=[batch_size, num_timesteps])[..., 0:3, :]
# place these into an ESM camera measurement container
esm_cam_meas = ESMCamMeasurement(
img_mean=img_mean,
cam_rel_mat=cam_rel_mat
)
# define agent pose transformation
agent_rel_mat = ivy.identity(4, batch_shape=[batch_size, num_timesteps])[..., 0:3, :]
# collect together into an ESM observation container
esm_obs = ESMObservation(
img_meas={'camera_0': esm_cam_meas},
agent_rel_mat=agent_rel_mat
)
# call model and test output
output = model(esm_obs)
assert output.shape[-1] == out_channels
Finally, we define a dummy loss function, and show how the ESM network can be trained using Ivy functions only.
.. code-block:: python
# define loss function
target = ivy.zeros_like(output)
def loss_fn(v):
pred = model(esm_obs, v=v)
return ivy.reduce_mean((pred - target) ** 2)
# optimizer
optimizer = ivy.SGD(lr=1e-4)
# train model
print('\ntraining dummy Ivy ESM model...\n')
for i in range(10):
loss, grads = ivy.execute_with_gradients(loss_fn, model.v)
model.v = optimizer.step(model.v, grads)
print('step {}, loss = {}'.format(i, ivy.to_numpy(loss).item()))
print('\ndummy Ivy ESM model trained!\n')
ivy.unset_framework()
**Neural Turing Machine**
We next show how the Ivy Neural Turing Machine (NTM) class can be used inside a TensorFlow model.
First, we define the model as below.
.. code-block:: python
class TfModelWithNTM(tf.keras.Model):
def __init__(self, channels_in, channels_out):
tf.keras.Model.__init__(self)
self._linear = tf.keras.layers.Dense(64)
memory_size = 4
memory_vector_dim = 1
self._ntm = ivy_mem.NTM(
input_dim=64, output_dim=channels_out, ctrl_output_size=channels_out, ctrl_layers=1,
memory_size=memory_size, memory_vector_dim=memory_vector_dim, read_head_num=1, write_head_num=1)
self._assign_variables()
def _assign_variables(self):
self._ntm.v.map(
lambda x, kc: self.add_weight(name=kc, shape=x.shape))
self.set_weights([ivy.to_numpy(v) for k, v in self._ntm.v.to_iterator()])
self.trainable_weights_dict = dict()
for weight in self.trainable_weights:
self.trainable_weights_dict[weight.name] = weight
self._ntm.v = self._ntm.v.map(lambda x, kc: self.trainable_weights_dict[kc + ':0'])
def call(self, x, **kwargs):
x = self._linear(x)
return self._ntm(x)
Next, we instantiate this model, and verify that the returned tensors are of the expected shape.
.. code-block:: python
# create model
in_channels = 32
out_channels = 8
ivy.set_framework('tensorflow')
model = TfModelWithNTM(in_channels, out_channels)
# define inputs
batch_shape = [1, 2]
timesteps = 3
input_shape = batch_shape + [timesteps, in_channels]
input_seq = tf.random.uniform(batch_shape + [timesteps, in_channels])
# call model and test output
output_seq = model(input_seq)
assert input_seq.shape[:-1] == output_seq.shape[:-1]
assert input_seq.shape[-1] == in_channels
assert output_seq.shape[-1] == out_channels
Finally, we define a dummy loss function, and show how the NTM can be trained using a native TensorFlow optimizer.
.. code-block:: python
# define loss function
target = tf.zeros_like(output_seq)
def loss_fn():
pred = model(input_seq)
return tf.reduce_sum((pred - target) ** 2)
# define optimizer
optimizer = tf.keras.optimizers.Adam(1e-2)
# train model
print('\ntraining dummy TensorFlow NTM model...\n')
for i in range(10):
with tf.GradientTape() as tape:
loss = loss_fn()
grads = tape.gradient(loss, model.trainable_weights)
optimizer.apply_gradients(zip(grads, model.trainable_weights))
print('step {}, loss = {}'.format(i, loss))
print('\ndummy TensorFlow NTM model trained!\n')
ivy.unset_framework()
Interactive Demos
-----------------
In addition to the run through above, we provide two further demo scripts,
which are more visual and interactive.
The scripts for these demos can be found in the interactive_ demos folder.
**Learning to Copy with NTM**
The first demo trains a Neural Turing Machine to copy a sequence from one memory bank to another.
NTM can overfit to a single copy sequence very quickly, as show in the real-time visualization below.
.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/ivy_memory/demo_a.png?raw=true
:width: 100%
**Mapping a Room with ESM**
The second demo creates an egocentric map of a room, from a rotating camera.
The raw image observations are shown on the left,
and the incrementally constructed omni-directional ESM feature and depth images are shown on the right.
While this example only projects color values into the memory, arbitrary neural network features can also be projected, for end-to-end training.
.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/ivy_memory/demo_b.png?raw=true
:width: 100%
Get Involed
-----------
We hope the memory classes in this library are useful to a wide range of machine learning developers.
However, there are many more areas of differentiable memory which could be covered by this library.
If there are any particular functions or classes you feel are missing,
and your needs are not met by the functions currently on offer,
then we are very happy to accept pull requests!
We look forward to working with the community on expanding and improving the Ivy memory library.
Citation
--------
::
@article{lenton2021ivy,
title={Ivy: Unified Machine Learning for Inter-Framework Portability},
author={Lenton, Daniel and Pardo, Fabio and Falck, Fabian and James, Stephen and Clark, Ronald},
journal={arXiv preprint arXiv:2102.02886},
year={2021}
}
Raw data
{
"_id": null,
"home_page": "https://ivy-dl.org/memory",
"name": "ivy-memory",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Ivy Team",
"author_email": "ivydl.team@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/85/ac/c82785b498247341329efb6dd042b8a92f00db3d90f0e12a35277e9b814d/ivy-memory-1.1.9.tar.gz",
"platform": "",
"description": ".. image:: https://github.com/ivy-dl/memory/blob/master/docs/partial_source/logos/logo.png?raw=true\n :width: 100%\n\n\n\n**End-to-end memory modules for machine learning developers, written in Ivy.**\n\n\n\n.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/logos/supported/frameworks.png?raw=true\n :width: 100%\n\nContents\n--------\n\n* `Overview`_\n* `Run Through`_\n* `Interactive Demos`_\n* `Get Involed`_\n\nOverview\n--------\n\n.. _docs: https://ivy-dl.org/memory\n\n**What is Ivy Memory?**\n\nIvy memory provides differentiable memory modules,\nincluding learnt modules such as Neural Turing Machines (NTM),\nbut also parameter-free modules such as End-to-End Egospheric Spatial Memory (ESM).\nCheck out the docs_ for more info!\n\nThe library is built on top of the Ivy machine learning framework.\nThis means all memory modules simultaneously support:\nJax, Tensorflow, PyTorch, MXNet, and Numpy.\n\n**Ivy Libraries**\n\nThere are a host of derived libraries written in Ivy, in the areas of mechanics, 3D vision, robotics, gym environments,\nneural memory, pre-trained models + implementations, and builder tools with trainers, data loaders and more. Click on\nthe icons below to learn more!\n\n\n\n.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/ivy_libraries.png?raw=true\n :width: 100%\n\n\n\n\n\n\n\n\n\n\n\n\n\n**Quick Start**\n\nIvy memory can be installed like so: ``pip install ivy-memory``\n\n.. _demos: https://github.com/ivy-dl/memory/tree/master/ivy_memory_demos\n.. _interactive: https://github.com/ivy-dl/memory/tree/master/ivy_memory_demos/interactive\n\nTo quickly see the different aspects of the library, we suggest you check out the demos_!\nWe suggest you start by running the script ``run_through.py``,\nand read the \"Run Through\" section below which explains this script.\n\nFor more interactive demos, we suggest you run either\n``learning_to_copy_with_ntm.py`` or ``mapping_a_room_with_esm.py`` in the interactive_ demos folder.\n\nRun Through\n-----------\n\nWe run through some of the different parts of the library via a simple ongoing example script.\nThe full script is available in the demos_ folder, as file ``run_through.py``.\n\n**End-to-End Egospheric Spatial Memory**\n\nFirst, we show how the Ivy End-to-End Egospheric Spatial Memory (ESM) class can be used inside a pure-Ivy model.\nWe first define the model as below.\n\n.. code-block:: python\n\n class IvyModelWithESM(ivy.Module):\n\n def __init__(self, channels_in, channels_out):\n self._channels_in = channels_in\n self._esm = ivy_mem.ESM(omni_image_dims=(16, 32))\n self._linear = ivy_mem.Linear(channels_in, channels_out)\n ivy.Module.__init__(self, 'cpu')\n\n def _forward(self, obs):\n mem = self._esm(obs)\n x = ivy.reshape(mem.mean, (-1, self._channels_in))\n return self._linear(x)\n\nNext, we instantiate this model, and verify that the returned tensors are of the expected shape.\n\n.. code-block:: python\n\n # create model\n in_channels = 32\n out_channels = 8\n ivy.set_framework('torch')\n model = IvyModelWithESM(in_channels, out_channels)\n\n # input config\n batch_size = 1\n image_dims = [5, 5]\n num_timesteps = 2\n num_feature_channels = 3\n\n # create image of pixel co-ordinates\n uniform_pixel_coords =\\\n ivy_vision.create_uniform_pixel_coords_image(image_dims, [batch_size, num_timesteps])\n\n # define camera measurement\n depths = ivy.random_uniform(shape=[batch_size, num_timesteps] + image_dims + [1])\n pixel_coords = ivy_vision.depth_to_pixel_coords(depths)\n inv_calib_mats = ivy.random_uniform(shape=[batch_size, num_timesteps, 3, 3])\n cam_coords = ivy_vision.pixel_to_cam_coords(pixel_coords, inv_calib_mats)[..., 0:3]\n features = ivy.random_uniform(shape=[batch_size, num_timesteps] + image_dims + [num_feature_channels])\n img_mean = ivy.concatenate((cam_coords, features), -1)\n cam_rel_mat = ivy.identity(4, batch_shape=[batch_size, num_timesteps])[..., 0:3, :]\n\n # place these into an ESM camera measurement container\n esm_cam_meas = ESMCamMeasurement(\n img_mean=img_mean,\n cam_rel_mat=cam_rel_mat\n )\n\n # define agent pose transformation\n agent_rel_mat = ivy.identity(4, batch_shape=[batch_size, num_timesteps])[..., 0:3, :]\n\n # collect together into an ESM observation container\n esm_obs = ESMObservation(\n img_meas={'camera_0': esm_cam_meas},\n agent_rel_mat=agent_rel_mat\n )\n\n # call model and test output\n output = model(esm_obs)\n assert output.shape[-1] == out_channels\n\nFinally, we define a dummy loss function, and show how the ESM network can be trained using Ivy functions only.\n\n.. code-block:: python\n\n # define loss function\n target = ivy.zeros_like(output)\n\n def loss_fn(v):\n pred = model(esm_obs, v=v)\n return ivy.reduce_mean((pred - target) ** 2)\n\n # optimizer\n optimizer = ivy.SGD(lr=1e-4)\n\n # train model\n print('\\ntraining dummy Ivy ESM model...\\n')\n for i in range(10):\n loss, grads = ivy.execute_with_gradients(loss_fn, model.v)\n model.v = optimizer.step(model.v, grads)\n print('step {}, loss = {}'.format(i, ivy.to_numpy(loss).item()))\n print('\\ndummy Ivy ESM model trained!\\n')\n ivy.unset_framework()\n\n**Neural Turing Machine**\n\nWe next show how the Ivy Neural Turing Machine (NTM) class can be used inside a TensorFlow model.\nFirst, we define the model as below.\n\n.. code-block:: python\n\n class TfModelWithNTM(tf.keras.Model):\n\n def __init__(self, channels_in, channels_out):\n tf.keras.Model.__init__(self)\n self._linear = tf.keras.layers.Dense(64)\n memory_size = 4\n memory_vector_dim = 1\n self._ntm = ivy_mem.NTM(\n input_dim=64, output_dim=channels_out, ctrl_output_size=channels_out, ctrl_layers=1,\n memory_size=memory_size, memory_vector_dim=memory_vector_dim, read_head_num=1, write_head_num=1)\n self._assign_variables()\n\n def _assign_variables(self):\n self._ntm.v.map(\n lambda x, kc: self.add_weight(name=kc, shape=x.shape))\n self.set_weights([ivy.to_numpy(v) for k, v in self._ntm.v.to_iterator()])\n self.trainable_weights_dict = dict()\n for weight in self.trainable_weights:\n self.trainable_weights_dict[weight.name] = weight\n self._ntm.v = self._ntm.v.map(lambda x, kc: self.trainable_weights_dict[kc + ':0'])\n\n def call(self, x, **kwargs):\n x = self._linear(x)\n return self._ntm(x)\n\nNext, we instantiate this model, and verify that the returned tensors are of the expected shape.\n\n.. code-block:: python\n\n # create model\n in_channels = 32\n out_channels = 8\n ivy.set_framework('tensorflow')\n model = TfModelWithNTM(in_channels, out_channels)\n\n # define inputs\n batch_shape = [1, 2]\n timesteps = 3\n input_shape = batch_shape + [timesteps, in_channels]\n input_seq = tf.random.uniform(batch_shape + [timesteps, in_channels])\n\n # call model and test output\n output_seq = model(input_seq)\n assert input_seq.shape[:-1] == output_seq.shape[:-1]\n assert input_seq.shape[-1] == in_channels\n assert output_seq.shape[-1] == out_channels\n\nFinally, we define a dummy loss function, and show how the NTM can be trained using a native TensorFlow optimizer.\n\n.. code-block:: python\n\n # define loss function\n target = tf.zeros_like(output_seq)\n\n def loss_fn():\n pred = model(input_seq)\n return tf.reduce_sum((pred - target) ** 2)\n\n # define optimizer\n optimizer = tf.keras.optimizers.Adam(1e-2)\n\n # train model\n print('\\ntraining dummy TensorFlow NTM model...\\n')\n for i in range(10):\n with tf.GradientTape() as tape:\n loss = loss_fn()\n grads = tape.gradient(loss, model.trainable_weights)\n optimizer.apply_gradients(zip(grads, model.trainable_weights))\n print('step {}, loss = {}'.format(i, loss))\n print('\\ndummy TensorFlow NTM model trained!\\n')\n ivy.unset_framework()\n\nInteractive Demos\n-----------------\n\nIn addition to the run through above, we provide two further demo scripts,\nwhich are more visual and interactive.\n\nThe scripts for these demos can be found in the interactive_ demos folder.\n\n**Learning to Copy with NTM**\n\nThe first demo trains a Neural Turing Machine to copy a sequence from one memory bank to another.\nNTM can overfit to a single copy sequence very quickly, as show in the real-time visualization below.\n\n\n\n.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/ivy_memory/demo_a.png?raw=true\n :width: 100%\n\n**Mapping a Room with ESM**\n\nThe second demo creates an egocentric map of a room, from a rotating camera.\nThe raw image observations are shown on the left,\nand the incrementally constructed omni-directional ESM feature and depth images are shown on the right.\nWhile this example only projects color values into the memory, arbitrary neural network features can also be projected, for end-to-end training.\n\n\n\n.. image:: https://github.com/ivy-dl/ivy-dl.github.io/blob/master/img/externally_linked/ivy_memory/demo_b.png?raw=true\n :width: 100%\n\nGet Involed\n-----------\n\nWe hope the memory classes in this library are useful to a wide range of machine learning developers.\nHowever, there are many more areas of differentiable memory which could be covered by this library.\n\nIf there are any particular functions or classes you feel are missing,\nand your needs are not met by the functions currently on offer,\nthen we are very happy to accept pull requests!\n\nWe look forward to working with the community on expanding and improving the Ivy memory library.\n\nCitation\n--------\n\n::\n\n @article{lenton2021ivy,\n title={Ivy: Unified Machine Learning for Inter-Framework Portability},\n author={Lenton, Daniel and Pardo, Fabio and Falck, Fabian and James, Stephen and Clark, Ronald},\n journal={arXiv preprint arXiv:2102.02886},\n year={2021}\n }\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "End-to-end memory modules for machine learning developers, written in Ivy.",
"version": "1.1.9",
"project_urls": {
"Docs": "https://ivy-dl.org/memory/",
"Homepage": "https://ivy-dl.org/memory",
"Source": "https://github.com/ivy-dl/memory"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "326ebfa3f3034ea0179749cd204d659bed88e1bce411c3773b29e46efe55d75b",
"md5": "3def849271ad364acd3778db1704f6f4",
"sha256": "568cd45d2cc3eed286a481b19afdc769b08d8799b51be80b45a472630182959d"
},
"downloads": -1,
"filename": "ivy_memory-1.1.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3def849271ad364acd3778db1704f6f4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 20739,
"upload_time": "2021-12-01T16:23:10",
"upload_time_iso_8601": "2021-12-01T16:23:10.006696Z",
"url": "https://files.pythonhosted.org/packages/32/6e/bfa3f3034ea0179749cd204d659bed88e1bce411c3773b29e46efe55d75b/ivy_memory-1.1.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "85acc82785b498247341329efb6dd042b8a92f00db3d90f0e12a35277e9b814d",
"md5": "87ff9047db58453dca35060513d1e061",
"sha256": "d0a7eeb219278dcf88146c315cb7c505bf91d5d7ad275aba6af6e6723f28584d"
},
"downloads": -1,
"filename": "ivy-memory-1.1.9.tar.gz",
"has_sig": false,
"md5_digest": "87ff9047db58453dca35060513d1e061",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 24748,
"upload_time": "2021-12-01T16:23:11",
"upload_time_iso_8601": "2021-12-01T16:23:11.173204Z",
"url": "https://files.pythonhosted.org/packages/85/ac/c82785b498247341329efb6dd042b8a92f00db3d90f0e12a35277e9b814d/ivy-memory-1.1.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2021-12-01 16:23:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ivy-dl",
"github_project": "memory",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "ivy-memory"
}
