<h1 style="color:green"><span style="color:green">Maté 🧉</span> your modular AI project and experiment manager</h1>
<!-- Maté is a python project, package and experiment manager. Whether you are a
seasoned deep learning researcher or just starting out, Maté provides you with
the tools to easily add source code and dependencies of models, trainers, and
data loaders to your projects. With Maté, you can also evaluate, train, and keep
track of your experiments with ease 🚀 -->
### Welcome to [Maté](https://github.com/oalee/yerbamate)! 🎉👋🏼
Maté is an open science modular Python framework designed to streamline and simplify the development and management of machine learning projects. It was developed to address the reproducibility crisis in artificial intelligence research by promoting open science and accessible AI. 🌍💻
The framework is built around the best software engineering practices of modularity and separation of concerns, encouraging quality coding, collaboration, and the sharing of models, trainers, data loaders, and knowledge. The modular design and separation of concerns simplify the development and maintenance of machine learning models, leading to an improved developer experience. 🚀💻👨💻💬🧠
With Maté, you can easily install the source code of open-source projects and adhere to modularity and separation of concerns, making your models and modules sharable out of the box. This means you can collaborate more effectively with others and easily share your work. 📦💻🤝
Thank you for choosing Maté, and we can't wait to see the amazing machine learning projects you'll create with it! 🎉👨💻🌟
## Features 🎉
- Seamless integration with any python library such as PyTorch/Lightning, TensorFlow/Keras, JAX/Flax, Huggingface/transformers. 🤝🤗🐉
- Easy to use interface to add source code of models, trainers, and data loaders to your projects. 🎨💻📝
- Support for full customizability and reproducibility of results through the inclusion of dependencies in your project. 🌟🔍🧪
- Modular project structure that enforces a clean and organized codebase. 🧱👨💻👌
- Fully compatible with python. No need to use mate commands to run your experiments. 🐍💻🚀
- Convenient environment management through the Maté Environment API. 🌍👨💻🔧
- Support for pip and conda for dependency management. 📦🔍💻
- Works with [Colab](https://github.com/oalee/yerbamate/blob/main/deep_learning.ipynb) out of the box. 🎉👌🤖
## Table of Contents
- [Installation](#installation-)
- [Quick Start](#quick-start-)
- [Project Structure](#project-structure-)
- [Example Projects](#example-projects-)
- [Documentation](#documentation-)
- [Contribution](#Contribution-)
- [Guides](#guides-)
- [Open Science](#open-science-)
## Installation 🔌
```bash
pip install yerbamate
```
## Quick Start ⚡
### **Initialize a project**
```bash
mate init deepnet
```
This will generate the following empty project structure:
```bash
/
|-- models/
| |-- __init__.py
|-- experiments/
| |-- __init__.py
|-- trainers/
| |-- __init__.py
|-- data/
| |-- __init__.py
```
### **Install an experiment**
To install an experiment, you can use `mate install` to install a module and its
dependencies from a github repository. See
[docs](https://oalee.github.io/yerbamate/#/?id=mate-install) for more details.
```bash
# Short version of GitHub URL https://github.com/oalee/big_transfer/tree/master/big_transfer/experiments/bit
mate install oalee/big_transfer/experiments/bit -yo pip
# Short version of GitHub URL https://github.com/oalee/deep-vision/tree/main/deepnet/experiments/resnet
mate install oalee/deep-vision/deepnet/experiments/resnet -yo pip
```
### **Install a module**
You can install independant modules such as models, trainers, and data loaders
from github projects that follow the
[Independent modular project structure](https://oalee.github.io/yerbamate/#/?id=project-structure-%f0%9f%93%81).
```bash
mate install oalee/lightweight-gan/lgan/trainers/lgan
mate install oalee/big_transfer/models/bit -yo pip
mate install oalee/deep-vision/deepnet/models/vit_pytorch -yo pip
mate install oalee/deep-vision/deepnet/trainers/classification -yo pip
```
### **Setting up environment**
Set up your environment before running your experiments. This can be done by using shell, or `env.json` file in the root of your project. Maté API requires `results` to be set in the environment. For more information, see [docs](https://oalee.github.io/yerbamate/#/?id=mat%c3%a9-environment-api).
```bash
DATA_PATH=/path/to/data
results=/path/to/results
```
```json
{
"DATA_PATH": "/path/to/data",
"results": "/path/to/results"
}
```
### **Train a model**
To train a model, you can use the `mate train` command. This command will train
the model with the specified experiment. For example, to train the an experiment
called `learn` in the `bit` module, you can use the following command:
```bash
mate train bit learn
# or alternatively use python
python -m deepnet.experiments.bit.learn train
```
## Project Structure 📁
Deep learning projects can be organized into the following structure with
modularity and seperation of concerns in mind. This offers a clean and organized
codebase that is easy to maintain and is sharable out-of-the-box.
The modular structure of the framework involves organizing the project directory in a hierarchical tree structure, with an arbitrary name given to the root project directory by the user. The project is then broken down into distinct concerns such as models, data, trainers, experiments, analyzers, and simulators, each with its own subdirectory. Within each concern, modules can be defined with their own subdirectories, such as specific models, trainers, data loaders, data augmentations, or loss functions.
```bash
└── project_name
├── data
│ ├── my_independent_data_loader
│ └── __init__.py
├── experiments
│ ├── my_awesome_experiment
│ └── __init__.py
├── __init__.py
├── models
│ ├── awesomenet
│ └── __init__.py
└── trainers
├── big_brain_trainer
└── __init__.py
```
### Modularity
Modularity is a software design principle that focuses on creating
self-contained, reusable and interchangeable components. In the context of a
deep learning project, modularity means creating three independent standalone
modules for models, trainers and data. This allows for a more organized,
maintainable and sharable project structure. The forth module, experiments, is
not independent, but rather combines the three modules together to create a
complete experiment.
### Independent Modules
Yerbamate prioritizes the organization of the project into independent modules when applicable. Independent modules only depend on Python dependencies (such as NumPy, PyTorch, TensorFlow, or Hugging Face), and the code inside the module uses relative imports to import within the module. This makes it an independent module that can be re-used when Python dependencies are installed.
### Non-Independent Modules
In some cases, a combination of independent modules may be necessary for a particular concern. An example of this is the experiment concern, which imports and combines models, data, and trainers to define and create a specific experiment. In such cases, the module is not independent and is designed to combine the previously defined independent modules. In the case of non-independent modules, Yerbamate creates a dependency list of independent modules that can be used to install the code and Python dependencies. This ensures that the necessary modules are installed, and that the code can be run without issue.
### Sample Modular Project Structure
This structure highlights modularity and seperation of concerns. The `models`,
`data` and `trainers` modules are independent and can be used in any project.
The `experiments` module is not independent, but rather combines the three
modules together to create a complete experiment.
```bash
.
├── mate.json
└── deepnet
├── data
│ ├── bit
│ │ ├── fewshot.py
│ │ ├── __init__.py
│ │ ├── minibatch_fewshot.py
│ │ ├── requirements.txt
│ │ └── transforms.py
│ └── __init__.py
├── experiments
│ ├── bit
│ │ ├── aug.py
│ │ ├── dependencies.json
│ │ ├── __init__.py
│ │ ├── learn.py
│ │ └── requirements.txt
│ └── __init__.py
├── __init__.py
├── models
│ ├── bit_torch
│ │ ├── downloader
│ │ │ ├── downloader.py
│ │ │ ├── __init__.py
│ │ │ ├── requirements.txt
│ │ │ └── utils.py
│ │ ├── __init__.py
│ │ ├── models.py
│ │ └── requirements.txt
│ └── __init__.py
└── trainers
├── bit_torch
│ ├── __init__.py
│ ├── lbtoolbox.py
│ ├── logger.py
│ ├── lr_schduler.py
│ ├── requirements.txt
│ └── trainer.py
└── __init__.py
```
## Example Projects 📚
Please check out the [transfer learning](https://github.com/oalee/big_transfer),
[vision models](https://github.com/oalee/deep-vision), and
[lightweight gan](https://github.com/oalee/lightweight-gan).
## Documentation 📚
Please check out the [documentation](https://oalee.github.io/yerbamate).
## Guides 📖
For more information on modularity, please check out this [guide](https://medium.com/@alee.rmi/the-ultimate-deep-learning-project-structure-a-software-engineers-guide-into-the-land-of-ai-c383f234fd2f).
For running experiments on Google Colab, please check out this
[example](https://github.com/oalee/yerbamate/blob/main/deep_learning.ipynb)
## Contribution 🤝
We welcome contributions from the community! Please check out our
[contributing](https://github.com/oalee/yerbamate/blob/main/CONTRIBUTING.md)
guide for more information on how to get started.
## Contact 🤝
For questions please contact:
oalee(at)proton.me
## Open Science 📖
As an open science work, Yerbamate strives to promote the principles of transparency and collaboration. To this end, the history of the LaTeX files for work are available on [GitHub](https://github.com/oalee/os-yerbamate). These open science repositories are open to collaboration and encourage participation from the community to enhance the validity, reproducibility, accessibility, and quality of this work.
Raw data
{
"_id": null,
"home_page": "https://github.com/oalee/yerbamate",
"name": "yerbamate",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "Giulio Zani, Ali Rahimi",
"author_email": "yerba.mate.dl@proton.me",
"download_url": "https://files.pythonhosted.org/packages/15/8d/f67fb7ee898e0ac26be2cbaa4bceb74b5501146e833335b55cc828dd16d4/yerbamate-0.9.239.tar.gz",
"platform": null,
"description": "<h1 style=\"color:green\"><span style=\"color:green\">Mat\u00e9 \ud83e\uddc9</span> your modular AI project and experiment manager</h1>\n\n<!-- Mat\u00e9 is a python project, package and experiment manager. Whether you are a\nseasoned deep learning researcher or just starting out, Mat\u00e9 provides you with\nthe tools to easily add source code and dependencies of models, trainers, and\ndata loaders to your projects. With Mat\u00e9, you can also evaluate, train, and keep\ntrack of your experiments with ease \ud83d\ude80 -->\n### Welcome to [Mat\u00e9](https://github.com/oalee/yerbamate)! \ud83c\udf89\ud83d\udc4b\ud83c\udffc\n\nMat\u00e9 is an open science modular Python framework designed to streamline and simplify the development and management of machine learning projects. It was developed to address the reproducibility crisis in artificial intelligence research by promoting open science and accessible AI. \ud83c\udf0d\ud83d\udcbb\n\nThe framework is built around the best software engineering practices of modularity and separation of concerns, encouraging quality coding, collaboration, and the sharing of models, trainers, data loaders, and knowledge. The modular design and separation of concerns simplify the development and maintenance of machine learning models, leading to an improved developer experience. \ud83d\ude80\ud83d\udcbb\ud83d\udc68\u200d\ud83d\udcbb\ud83d\udcac\ud83e\udde0\n\nWith Mat\u00e9, you can easily install the source code of open-source projects and adhere to modularity and separation of concerns, making your models and modules sharable out of the box. This means you can collaborate more effectively with others and easily share your work. \ud83d\udce6\ud83d\udcbb\ud83e\udd1d\n\nThank you for choosing Mat\u00e9, and we can't wait to see the amazing machine learning projects you'll create with it! \ud83c\udf89\ud83d\udc68\u200d\ud83d\udcbb\ud83c\udf1f\n\n## Features \ud83c\udf89\n- Seamless integration with any python library such as PyTorch/Lightning, TensorFlow/Keras, JAX/Flax, Huggingface/transformers. \ud83e\udd1d\ud83e\udd17\ud83d\udc09\n- Easy to use interface to add source code of models, trainers, and data loaders to your projects. \ud83c\udfa8\ud83d\udcbb\ud83d\udcdd\n- Support for full customizability and reproducibility of results through the inclusion of dependencies in your project. \ud83c\udf1f\ud83d\udd0d\ud83e\uddea\n- Modular project structure that enforces a clean and organized codebase. \ud83e\uddf1\ud83d\udc68\u200d\ud83d\udcbb\ud83d\udc4c\n- Fully compatible with python. No need to use mate commands to run your experiments. \ud83d\udc0d\ud83d\udcbb\ud83d\ude80\n- Convenient environment management through the Mat\u00e9 Environment API. \ud83c\udf0d\ud83d\udc68\u200d\ud83d\udcbb\ud83d\udd27\n- Support for pip and conda for dependency management. \ud83d\udce6\ud83d\udd0d\ud83d\udcbb\n- Works with [Colab](https://github.com/oalee/yerbamate/blob/main/deep_learning.ipynb) out of the box. \ud83c\udf89\ud83d\udc4c\ud83e\udd16\n\n## Table of Contents\n\n- [Installation](#installation-)\n- [Quick Start](#quick-start-)\n- [Project Structure](#project-structure-)\n- [Example Projects](#example-projects-)\n- [Documentation](#documentation-)\n- [Contribution](#Contribution-)\n- [Guides](#guides-)\n- [Open Science](#open-science-)\n\n## Installation \ud83d\udd0c\n\n```bash\npip install yerbamate\n```\n\n## Quick Start \u26a1\n\n### **Initialize a project**\n\n```bash\nmate init deepnet\n```\n\nThis will generate the following empty project structure:\n\n```bash\n/\n|-- models/\n| |-- __init__.py\n|-- experiments/\n| |-- __init__.py\n|-- trainers/\n| |-- __init__.py\n|-- data/\n| |-- __init__.py\n```\n\n### **Install an experiment**\n\nTo install an experiment, you can use `mate install` to install a module and its\ndependencies from a github repository. See\n[docs](https://oalee.github.io/yerbamate/#/?id=mate-install) for more details.\n\n```bash\n# Short version of GitHub URL https://github.com/oalee/big_transfer/tree/master/big_transfer/experiments/bit\nmate install oalee/big_transfer/experiments/bit -yo pip\n\n# Short version of GitHub URL https://github.com/oalee/deep-vision/tree/main/deepnet/experiments/resnet\nmate install oalee/deep-vision/deepnet/experiments/resnet -yo pip\n```\n\n### **Install a module**\n\nYou can install independant modules such as models, trainers, and data loaders\nfrom github projects that follow the\n[Independent modular project structure](https://oalee.github.io/yerbamate/#/?id=project-structure-%f0%9f%93%81).\n\n```bash\nmate install oalee/lightweight-gan/lgan/trainers/lgan \nmate install oalee/big_transfer/models/bit -yo pip\nmate install oalee/deep-vision/deepnet/models/vit_pytorch -yo pip\nmate install oalee/deep-vision/deepnet/trainers/classification -yo pip\n```\n\n### **Setting up environment**\n\nSet up your environment before running your experiments. This can be done by using shell, or `env.json` file in the root of your project. Mat\u00e9 API requires `results` to be set in the environment. For more information, see [docs](https://oalee.github.io/yerbamate/#/?id=mat%c3%a9-environment-api).\n```bash\nDATA_PATH=/path/to/data\nresults=/path/to/results\n```\n```json\n{\n \"DATA_PATH\": \"/path/to/data\",\n \"results\": \"/path/to/results\"\n}\n```\n\n### **Train a model**\n\nTo train a model, you can use the `mate train` command. This command will train\nthe model with the specified experiment. For example, to train the an experiment\ncalled `learn` in the `bit` module, you can use the following command:\n\n```bash\nmate train bit learn\n# or alternatively use python\npython -m deepnet.experiments.bit.learn train\n```\n\n## Project Structure \ud83d\udcc1\n\nDeep learning projects can be organized into the following structure with\nmodularity and seperation of concerns in mind. This offers a clean and organized\ncodebase that is easy to maintain and is sharable out-of-the-box.\nThe modular structure of the framework involves organizing the project directory in a hierarchical tree structure, with an arbitrary name given to the root project directory by the user. The project is then broken down into distinct concerns such as models, data, trainers, experiments, analyzers, and simulators, each with its own subdirectory. Within each concern, modules can be defined with their own subdirectories, such as specific models, trainers, data loaders, data augmentations, or loss functions.\n\n```bash\n\u2514\u2500\u2500 project_name\n \u251c\u2500\u2500 data\n \u2502 \u251c\u2500\u2500 my_independent_data_loader\n \u2502 \u2514\u2500\u2500 __init__.py\n \u251c\u2500\u2500 experiments\n \u2502 \u251c\u2500\u2500 my_awesome_experiment\n \u2502 \u2514\u2500\u2500 __init__.py\n \u251c\u2500\u2500 __init__.py\n \u251c\u2500\u2500 models\n \u2502 \u251c\u2500\u2500 awesomenet\n \u2502 \u2514\u2500\u2500 __init__.py\n \u2514\u2500\u2500 trainers\n \u251c\u2500\u2500 big_brain_trainer\n \u2514\u2500\u2500 __init__.py\n```\n\n### Modularity\n\nModularity is a software design principle that focuses on creating\nself-contained, reusable and interchangeable components. In the context of a\ndeep learning project, modularity means creating three independent standalone\nmodules for models, trainers and data. This allows for a more organized,\nmaintainable and sharable project structure. The forth module, experiments, is\nnot independent, but rather combines the three modules together to create a\ncomplete experiment.\n\n### Independent Modules\n\nYerbamate prioritizes the organization of the project into independent modules when applicable. Independent modules only depend on Python dependencies (such as NumPy, PyTorch, TensorFlow, or Hugging Face), and the code inside the module uses relative imports to import within the module. This makes it an independent module that can be re-used when Python dependencies are installed.\n\n### Non-Independent Modules\nIn some cases, a combination of independent modules may be necessary for a particular concern. An example of this is the experiment concern, which imports and combines models, data, and trainers to define and create a specific experiment. In such cases, the module is not independent and is designed to combine the previously defined independent modules. In the case of non-independent modules, Yerbamate creates a dependency list of independent modules that can be used to install the code and Python dependencies. This ensures that the necessary modules are installed, and that the code can be run without issue.\n\n\n### Sample Modular Project Structure\n\nThis structure highlights modularity and seperation of concerns. The `models`,\n`data` and `trainers` modules are independent and can be used in any project.\nThe `experiments` module is not independent, but rather combines the three\nmodules together to create a complete experiment.\n\n```bash\n.\n\u251c\u2500\u2500 mate.json\n\u2514\u2500\u2500 deepnet\n \u251c\u2500\u2500 data\n \u2502 \u251c\u2500\u2500 bit\n \u2502 \u2502 \u251c\u2500\u2500 fewshot.py\n \u2502 \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u2502 \u251c\u2500\u2500 minibatch_fewshot.py\n \u2502 \u2502 \u251c\u2500\u2500 requirements.txt\n \u2502 \u2502 \u2514\u2500\u2500 transforms.py\n \u2502 \u2514\u2500\u2500 __init__.py\n \u251c\u2500\u2500 experiments\n \u2502 \u251c\u2500\u2500 bit\n \u2502 \u2502 \u251c\u2500\u2500 aug.py\n \u2502 \u2502 \u251c\u2500\u2500 dependencies.json\n \u2502 \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u2502 \u251c\u2500\u2500 learn.py\n \u2502 \u2502 \u2514\u2500\u2500 requirements.txt\n \u2502 \u2514\u2500\u2500 __init__.py\n \u251c\u2500\u2500 __init__.py\n \u251c\u2500\u2500 models\n \u2502 \u251c\u2500\u2500 bit_torch\n \u2502 \u2502 \u251c\u2500\u2500 downloader\n \u2502 \u2502 \u2502 \u251c\u2500\u2500 downloader.py\n \u2502 \u2502 \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u2502 \u2502 \u251c\u2500\u2500 requirements.txt\n \u2502 \u2502 \u2502 \u2514\u2500\u2500 utils.py\n \u2502 \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u2502 \u251c\u2500\u2500 models.py\n \u2502 \u2502 \u2514\u2500\u2500 requirements.txt\n \u2502 \u2514\u2500\u2500 __init__.py\n \u2514\u2500\u2500 trainers\n \u251c\u2500\u2500 bit_torch\n \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u251c\u2500\u2500 lbtoolbox.py\n \u2502 \u251c\u2500\u2500 logger.py\n \u2502 \u251c\u2500\u2500 lr_schduler.py\n \u2502 \u251c\u2500\u2500 requirements.txt\n \u2502 \u2514\u2500\u2500 trainer.py\n \u2514\u2500\u2500 __init__.py\n```\n\n## Example Projects \ud83d\udcda\n\nPlease check out the [transfer learning](https://github.com/oalee/big_transfer),\n[vision models](https://github.com/oalee/deep-vision), and\n[lightweight gan](https://github.com/oalee/lightweight-gan).\n\n## Documentation \ud83d\udcda\n\nPlease check out the [documentation](https://oalee.github.io/yerbamate).\n\n## Guides \ud83d\udcd6\n\nFor more information on modularity, please check out this [guide](https://medium.com/@alee.rmi/the-ultimate-deep-learning-project-structure-a-software-engineers-guide-into-the-land-of-ai-c383f234fd2f).\n\nFor running experiments on Google Colab, please check out this\n[example](https://github.com/oalee/yerbamate/blob/main/deep_learning.ipynb)\n\n## Contribution \ud83e\udd1d\n\nWe welcome contributions from the community! Please check out our\n[contributing](https://github.com/oalee/yerbamate/blob/main/CONTRIBUTING.md)\nguide for more information on how to get started.\n\n## Contact \ud83e\udd1d\n\nFor questions please contact:\n\noalee(at)proton.me\n\n\n## Open Science \ud83d\udcd6\n\nAs an open science work, Yerbamate strives to promote the principles of transparency and collaboration. To this end, the history of the LaTeX files for work are available on [GitHub](https://github.com/oalee/os-yerbamate). These open science repositories are open to collaboration and encourage participation from the community to enhance the validity, reproducibility, accessibility, and quality of this work.\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A python module and experiment manager for deep learning",
"version": "0.9.239",
"project_urls": {
"Homepage": "https://github.com/oalee/yerbamate"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "158df67fb7ee898e0ac26be2cbaa4bceb74b5501146e833335b55cc828dd16d4",
"md5": "7f99cba0d587eb7abfb7ecffa888f3c6",
"sha256": "96a57c94da81d104c43ca19a825e6c74df911a87fc9951411b8b78271656c233"
},
"downloads": -1,
"filename": "yerbamate-0.9.239.tar.gz",
"has_sig": false,
"md5_digest": "7f99cba0d587eb7abfb7ecffa888f3c6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 66219,
"upload_time": "2023-05-13T17:53:27",
"upload_time_iso_8601": "2023-05-13T17:53:27.282967Z",
"url": "https://files.pythonhosted.org/packages/15/8d/f67fb7ee898e0ac26be2cbaa4bceb74b5501146e833335b55cc828dd16d4/yerbamate-0.9.239.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-13 17:53:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "oalee",
"github_project": "yerbamate",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "yerbamate"
}
JSON
