# lightning-GPT
lightning-GPT is a minimal wrapper around Andrej Karpathy's [minGPT](https://github.com/karpathy/minGPT) and [nanoGPT](https://github.com/karpathy/nanoGPT) in Lightning.
It is aimed at providing a minimal Lightning layer on top of minGPT and nanoGPT, while leveraging the full breadth of Lightning.
There are currently a few options:
- `MinGPT`: the GPT model from minGPT vanilla (set `--implementation=mingpt`)
- `NanoGPT`: the GPT model from nanoGPT vanilla (set `--implementation=nanogpt`)
- `DeepSpeedMinGPT`: the GPT model from minGPT made DeepSpeed-ready (set `--strategy=deepspeed`)
- `DeepSpeedNanoGPT`: the GPT model from nanoGPT made DeepSpeed-ready (set `--strategy=deepspeed`)
- `FSDPMinGPT`: the GPT model from minGPT made FSDP (native)-ready (set `--strategy=fsdp-gpt`)
- `FSDPNanoGPT`: the GPT model from nanoGPT made FSDP (native)-ready (set `--strategy=fsdp-gpt`)
minGPT and nanoGPT are vendored with the repo in the `mingpt` and `nanogpt` directories respectively. Find the respective LICENSE there.
Thanks to:
- @karpathy for the original minGPT and nanoGPT implementation
- @williamFalcon for the first Lightning port
- @SeanNaren for the DeepSpeed pieces
## Installation
There are two main ways to install this package.
Installation from source is preferred if you need the latest version with yet unreleased changes, want to use the provided benchmarking or training suites or need to adjust the package.
Installation from PyPI is preferred if you just want to use a stable version of the package without any modifications.
### Installation from PyPI
To install the package, simply run
```shell
pip install lightning-gpt
```
### Installation from source
To clone the repository, please clone the repo with
```shell
git clone https://github.com/Lightning-AI/lightning-GPT && cd lightning-GPT
git submodule update --init --recursive
```
and install with
```shell
pip install -e .
```
After this you can proceed with the following steps.
## MinGPT
First install the dependencies
```shell
pip install -r requirements.txt
```
then
```shell
python train.py
```
See
```shell
python train.py --help
```
for the available flags.
## NanoGPT
First install the dependencies.
```shell
pip install -r requirements.txt
pip install -r requirements/nanogpt.txt
```
then
```shell
python train.py
```
See
```shell
python train.py --help
```
for the available flags.
## DeepSpeed
Install the extra-dependencies:
```shell
pip install -r requirements/deepspeed.txt
```
and pass the `strategy` flag to the script
```shell
python train.py --implementation mingpt --strategy deepspeed
```
or
```shell
python train.py --implementation nanogpt --strategy deepspeed
```
## FSDP native
Pass the `strategy` flag to the script
```shell
python train.py --implementation mingpt --strategy fsdp_native
```
or
```shell
python train.py --implementation nanogpt --strategy fsdp_native
```
## PyTorch 2.0
To run on dynamo/inductor from the PyTorch 2.0 compiler stack, run
```shell
python train.py --compile dynamo
```
Note that you will need a recent `torch` nightly (1.14.x) for `torch.compile`
to be available.
## Credits
- https://github.com/karpathy/nanoGPT
- https://github.com/karpathy/minGPT
- https://github.com/SeanNaren/minGPT
- https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html
## License
Apache 2.0 license https://opensource.org/licenses/Apache-2.0
Raw data
{
"_id": null,
"home_page": "https://github.com/Lightning-AI/lightning-GPT",
"name": "lightning-gpt",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "deep learning,machine learning,pytorch,transformers,gpt,AI",
"author": "Lightning-AI et al.",
"author_email": "luca@lightning.ai",
"download_url": "https://files.pythonhosted.org/packages/35/fd/eaf9c5d7b3934e4ee504f5cf778ada80c5e1c242b6092ffabf2ba90ce2cc/lightning_gpt-0.1.1.tar.gz",
"platform": null,
"description": "# lightning-GPT\n\nlightning-GPT is a minimal wrapper around Andrej Karpathy's [minGPT](https://github.com/karpathy/minGPT) and [nanoGPT](https://github.com/karpathy/nanoGPT) in Lightning.\n\nIt is aimed at providing a minimal Lightning layer on top of minGPT and nanoGPT, while leveraging the full breadth of Lightning.\n\nThere are currently a few options:\n\n- `MinGPT`: the GPT model from minGPT vanilla (set `--implementation=mingpt`)\n- `NanoGPT`: the GPT model from nanoGPT vanilla (set `--implementation=nanogpt`)\n- `DeepSpeedMinGPT`: the GPT model from minGPT made DeepSpeed-ready (set `--strategy=deepspeed`)\n- `DeepSpeedNanoGPT`: the GPT model from nanoGPT made DeepSpeed-ready (set `--strategy=deepspeed`)\n- `FSDPMinGPT`: the GPT model from minGPT made FSDP (native)-ready (set `--strategy=fsdp-gpt`)\n- `FSDPNanoGPT`: the GPT model from nanoGPT made FSDP (native)-ready (set `--strategy=fsdp-gpt`)\n\nminGPT and nanoGPT are vendored with the repo in the `mingpt` and `nanogpt` directories respectively. Find the respective LICENSE there.\n\nThanks to:\n\n- @karpathy for the original minGPT and nanoGPT implementation\n- @williamFalcon for the first Lightning port\n- @SeanNaren for the DeepSpeed pieces\n\n## Installation\n\nThere are two main ways to install this package.\n\nInstallation from source is preferred if you need the latest version with yet unreleased changes, want to use the provided benchmarking or training suites or need to adjust the package.\n\nInstallation from PyPI is preferred if you just want to use a stable version of the package without any modifications.\n\n### Installation from PyPI\n\nTo install the package, simply run\n\n```shell\npip install lightning-gpt\n```\n\n### Installation from source\n\nTo clone the repository, please clone the repo with\n\n```shell\ngit clone https://github.com/Lightning-AI/lightning-GPT && cd lightning-GPT\ngit submodule update --init --recursive\n```\n\nand install with\n\n```shell\npip install -e .\n```\n\nAfter this you can proceed with the following steps.\n\n## MinGPT\n\nFirst install the dependencies\n\n```shell\npip install -r requirements.txt\n```\n\nthen\n\n```shell\npython train.py\n```\n\nSee\n\n```shell\npython train.py --help\n```\n\nfor the available flags.\n\n## NanoGPT\n\nFirst install the dependencies.\n\n```shell\npip install -r requirements.txt\npip install -r requirements/nanogpt.txt\n```\n\nthen\n\n```shell\npython train.py\n```\n\nSee\n\n```shell\npython train.py --help\n```\n\nfor the available flags.\n\n## DeepSpeed\n\nInstall the extra-dependencies:\n\n```shell\npip install -r requirements/deepspeed.txt\n```\n\nand pass the `strategy` flag to the script\n\n```shell\npython train.py --implementation mingpt --strategy deepspeed\n```\n\nor\n\n```shell\npython train.py --implementation nanogpt --strategy deepspeed\n```\n\n## FSDP native\n\nPass the `strategy` flag to the script\n\n```shell\npython train.py --implementation mingpt --strategy fsdp_native\n```\n\nor\n\n```shell\npython train.py --implementation nanogpt --strategy fsdp_native\n```\n\n## PyTorch 2.0\n\nTo run on dynamo/inductor from the PyTorch 2.0 compiler stack, run\n\n```shell\npython train.py --compile dynamo\n```\n\nNote that you will need a recent `torch` nightly (1.14.x) for `torch.compile`\nto be available.\n\n## Credits\n\n- https://github.com/karpathy/nanoGPT\n- https://github.com/karpathy/minGPT\n- https://github.com/SeanNaren/minGPT\n- https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html\n\n## License\n\nApache 2.0 license https://opensource.org/licenses/Apache-2.0\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "GPT training in Lightning",
"version": "0.1.1",
"project_urls": {
"Bug Tracker": "https://github.com/Lightning-AI/lightning-GPT/issues",
"Documentation": "https://github.com/Lightning-AI/lightning-GPT",
"Download": "https://github.com/Lightning-AI/lightning-GPT/archive/main.zip",
"Homepage": "https://github.com/Lightning-AI/lightning-GPT",
"Source Code": "https://github.com/Lightning-AI/lightning-GPT"
},
"split_keywords": [
"deep learning",
"machine learning",
"pytorch",
"transformers",
"gpt",
"ai"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5c2a7ea9993d758899cd33d4b5faa73eff933f7034dc2b1ce4b5a7e5e099ca86",
"md5": "454c41460c0598e63513f6b1c87a40da",
"sha256": "aa450e87f4716145d8a97fbbb838dc5fc380647e728e3acb218d7b3042285487"
},
"downloads": -1,
"filename": "lightning_gpt-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "454c41460c0598e63513f6b1c87a40da",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 466554,
"upload_time": "2023-11-01T12:23:32",
"upload_time_iso_8601": "2023-11-01T12:23:32.233702Z",
"url": "https://files.pythonhosted.org/packages/5c/2a/7ea9993d758899cd33d4b5faa73eff933f7034dc2b1ce4b5a7e5e099ca86/lightning_gpt-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "35fdeaf9c5d7b3934e4ee504f5cf778ada80c5e1c242b6092ffabf2ba90ce2cc",
"md5": "e7afb174645a5da45512d6badacfb2e5",
"sha256": "514800fdce06a9a01fe29095fd2a60075010eda1ff5e6e4e8a563b8e6a2cd3a1"
},
"downloads": -1,
"filename": "lightning_gpt-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "e7afb174645a5da45512d6badacfb2e5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 595867,
"upload_time": "2023-11-01T12:23:33",
"upload_time_iso_8601": "2023-11-01T12:23:33.849212Z",
"url": "https://files.pythonhosted.org/packages/35/fd/eaf9c5d7b3934e4ee504f5cf778ada80c5e1c242b6092ffabf2ba90ce2cc/lightning_gpt-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-01 12:23:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Lightning-AI",
"github_project": "lightning-GPT",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "lightning-gpt"
}
JSON
