| Name | crafter JSON |
| Version |
1.8.2
JSON |
| download |
| home_page | http://github.com/danijar/crafter |
| Summary | Open world survival game for reinforcement learning. |
| upload_time | 2023-10-31 00:46:32 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | |
| license | |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
**Status:** Stable release
[](https://pypi.python.org/pypi/crafter/#history)
# Crafter
Open world survival game for evaluating a wide range of agent abilities within
a single environment.
## Overview
Crafter features randomly generated 2D worlds where the player needs to forage
for food and water, find shelter to sleep, defend against monsters, collect
materials, and build tools. Crafter aims to be a fruitful benchmark for
reinforcement learning by focusing on the following design goals:
- **Research challenges:** Crafter poses substantial challenges to current
methods, evaluating strong generalization, wide and deep exploration,
representation learning, and long-term reasoning and credit assignment.
- **Meaningful evaluation:** Agents are evaluated by semantically meaningful
achievements that can be unlocked in each episode, offering insights into the
ability spectrum of both reward agents and unsupervised agents.
- **Iteration speed:** Crafter evaluates many agent abilities within a single
env, vastly reducing the computational requirements over benchmarks suites
that require training on many separate envs from scratch.
See the research paper to find out more: [Benchmarking the Spectrum of Agent
Capabilities](https://arxiv.org/pdf/2109.06780.pdf)
```
@article{hafner2021crafter,
title={Benchmarking the Spectrum of Agent Capabilities},
author={Danijar Hafner},
year={2021},
journal={arXiv preprint arXiv:2109.06780},
}
```
## Play Yourself
```sh
python3 -m pip install crafter # Install Crafter
python3 -m pip install pygame # Needed for human interface
python3 -m crafter.run_gui # Start the game
```
<details>
<summary>Keyboard mapping (click to expand)</summary>
| Key | Action |
| :-: | :----- |
| WASD | Move around |
| SPACE| Collect material, drink from lake, hit creature |
| TAB | Sleep |
| T | Place a table |
| R | Place a rock |
| F | Place a furnace |
| P | Place a plant |
| 1 | Craft a wood pickaxe |
| 2 | Craft a stone pickaxe |
| 3 | Craft an iron pickaxe |
| 4 | Craft a wood sword |
| 5 | Craft a stone sword |
| 6 | Craft an iron sword |
</details>
## Interface
To install Crafter, run `pip3 install crafter`. The environment follows the
[OpenAI Gym][gym] interface. Observations are images of size (64, 64, 3) and
outputs are one of 17 categorical actions.
```py
import gym
import crafter
env = gym.make('CrafterReward-v1') # Or CrafterNoReward-v1
env = crafter.Recorder(
env, './path/to/logdir',
save_stats=True,
save_video=False,
save_episode=False,
)
obs = env.reset()
done = False
while not done:
action = env.action_space.sample()
obs, reward, done, info = env.step(action)
```
[gym]: https://github.com/openai/gym
## Evaluation
Agents are allowed a budget of 1M environmnent steps and are evaluated by their
success rates of the 22 achievements and by their geometric mean score. Example
scripts for computing these are included in the `analysis` directory of the
repository.
- **Reward:** The sparse reward is `+1` for unlocking an achievement during
the episode and `-0.1` or `+0.1` for lost or regenerated health points.
Results should be reported not as reward but as success rates and score.
- **Success rates:** The success rates of the 22 achievemnts are computed
as the percentage across all training episodes in which the achievement was
unlocked, allowing insights into the ability spectrum of an agent.
- **Crafter score:** The score is the geometric mean of success rates, so that
improvements on difficult achievements contribute more than improvements on
achievements with already high success rates.
## Scoreboards
Please create a pull request if you would like to add your or another algorithm
to the scoreboards. For the reinforcement learning and unsupervised agents
categories, the interaction budget is 1M. The external knowledge category is
defined more broadly.
### Reinforcement Learning
| Algorithm | Score (%) | Reward | Open Source |
|:----------|----------:|-------:|:-----------:|
| [Curious Replay](https://arxiv.org/pdf/2306.15934.pdf) | 19.4±1.6 | - | [AutonomousAgentsLab/cr-dv3](https://github.com/AutonomousAgentsLab/cr-dv3) |
| [PPO (ResNet)](https://arxiv.org/pdf/2307.03486.pdf)| 15.6±1.6 | 10.3±0.5 | [snu-mllab/Achievement-Distillation](https://github.com/snu-mllab/Achievement-Distillation)
| [DreamerV3](https://arxiv.org/pdf/2301.04104v1.pdf) | 14.5±1.6 | 11.7±1.9 | [danijar/dreamerv3](https://github.com/danijar/dreamerv3) |
| [LSTM-SPCNN](https://arxiv.org/pdf/2208.03374.pdf) | 12.1±0.8 | — | [astanic/crafter-ood](https://github.com/astanic/crafter-ood) |
| [EDE](https://openreview.net/pdf?id=GZDsKahGY-2) | 11.7±1.0 | — | [yidingjiang/ede](https://github.com/yidingjiang/ede) |
| [OC-SA](https://arxiv.org/pdf/2208.03374.pdf) | 11.1±0.7 | — | [astanic/crafter-ood](https://github.com/astanic/crafter-ood) |
| [DreamerV2](https://arxiv.org/pdf/2010.02193.pdf) | 10.0±1.2 | 9.0±1.7 | [danijar/dreamerv2](https://github.com/danijar/dreamerv2) |
| [PPO](https://arxiv.org/pdf/1710.02298.pdf) | 4.6±0.3 | 4.2±1.2 | [DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3) |
| [Rainbow](https://arxiv.org/pdf/1710.02298.pdf) | 4.3±0.2 | 6.0±1.3 | [Kaixhin/Rainbow](https://github.com/Kaixhin/Rainbow) |
### Unsupervised Agents
| Algorithm | Score (%) | Reward | Open Source |
|:----------|----------:|-------:|:-----------:|
| [Plan2Explore](https://arxiv.org/pdf/2010.02193.pdf) | 2.1±0.1 | 2.1±1.5 | [danijar/dreamerv2](https://github.com/danijar/dreamerv2) |
| [RND](https://arxiv.org/pdf/1810.12894.pdf) | 2.0±0.1 | 0.7±1.3 | [alirezakazemipour/PPO-RND](https://github.com/alirezakazemipour/PPO-RND) |
| Random | 1.6±0.0 | 2.1±1.3 | — |
### External Knowledge
| Algorithm | Score (%) | Reward | Uses | Interaction | Open Source |
|:----------|----------:|-------:|:-----|:-----------:|:-----------:|
| [Human](https://en.wikipedia.org/wiki/Human) | 50.5±6.8 | 14.3±2.3 | Life experience | 0 | [crafter_human_dataset](https://archive.org/details/crafter_human_dataset) |
| [SPRING](https://arxiv.org/pdf/2305.15486.pdf) | 27.3±1.2 | 12.3±0.7 | LLM, scene description, Crafter paper | 0 | ❌ |
| [Achievement Distillation](https://arxiv.org/pdf/2307.03486.pdf) | 21.8±1.4 | 12.6±0.3 | Reward structure | 1M | [snu-mllab/Achievement-Distillation](https://github.com/snu-mllab/Achievement-Distillation) |
| [ELLM](https://arxiv.org/pdf/2302.06692.pdf) | — | 6.0±0.4 | LLM, scene description | 5M | ❌ |
## Baselines
Baseline scores of various agents are available for Crafter, both with and
without rewards. The scores are available in JSON format in the `scores`
directory of the repository. For comparison, the score of human expert players
is 50.5\%. The [baseline
implementations](https://github.com/danijar/crafter-baselines) are available as
a separate repository.
<img src="https://github.com/danijar/crafter/raw/main/media/scores.png" width="400"/>
## Questions
Please [open an issue][issues] on Github.
[issues]: https://github.com/danijar/crafter/issues
Raw data
{
"_id": null,
"home_page": "http://github.com/danijar/crafter",
"name": "crafter",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/2d/2a/39dcc19284758fd8502ec2dad4cbde25a32cf51d33c72d76efa9cb081fa4/crafter-1.8.2.tar.gz",
"platform": null,
"description": "**Status:** Stable release\n\n[](https://pypi.python.org/pypi/crafter/#history)\n\n# Crafter\n\nOpen world survival game for evaluating a wide range of agent abilities within\na single environment.\n\n\n\n## Overview\n\nCrafter features randomly generated 2D worlds where the player needs to forage\nfor food and water, find shelter to sleep, defend against monsters, collect\nmaterials, and build tools. Crafter aims to be a fruitful benchmark for\nreinforcement learning by focusing on the following design goals:\n\n- **Research challenges:** Crafter poses substantial challenges to current\n methods, evaluating strong generalization, wide and deep exploration,\n representation learning, and long-term reasoning and credit assignment.\n\n- **Meaningful evaluation:** Agents are evaluated by semantically meaningful\n achievements that can be unlocked in each episode, offering insights into the\n ability spectrum of both reward agents and unsupervised agents.\n\n- **Iteration speed:** Crafter evaluates many agent abilities within a single\n env, vastly reducing the computational requirements over benchmarks suites\n that require training on many separate envs from scratch.\n\nSee the research paper to find out more: [Benchmarking the Spectrum of Agent\nCapabilities](https://arxiv.org/pdf/2109.06780.pdf)\n\n```\n@article{hafner2021crafter,\n title={Benchmarking the Spectrum of Agent Capabilities},\n author={Danijar Hafner},\n year={2021},\n journal={arXiv preprint arXiv:2109.06780},\n}\n```\n\n## Play Yourself\n\n```sh\npython3 -m pip install crafter # Install Crafter\npython3 -m pip install pygame # Needed for human interface\npython3 -m crafter.run_gui # Start the game\n```\n\n<details>\n<summary>Keyboard mapping (click to expand)</summary>\n\n| Key | Action |\n| :-: | :----- |\n| WASD | Move around |\n| SPACE| Collect material, drink from lake, hit creature |\n| TAB | Sleep |\n| T | Place a table |\n| R | Place a rock |\n| F | Place a furnace |\n| P | Place a plant |\n| 1 | Craft a wood pickaxe |\n| 2 | Craft a stone pickaxe |\n| 3 | Craft an iron pickaxe |\n| 4 | Craft a wood sword |\n| 5 | Craft a stone sword |\n| 6 | Craft an iron sword |\n\n</details>\n\n\n\n## Interface\n\nTo install Crafter, run `pip3 install crafter`. The environment follows the\n[OpenAI Gym][gym] interface. Observations are images of size (64, 64, 3) and\noutputs are one of 17 categorical actions.\n\n```py\nimport gym\nimport crafter\n\nenv = gym.make('CrafterReward-v1') # Or CrafterNoReward-v1\nenv = crafter.Recorder(\n env, './path/to/logdir',\n save_stats=True,\n save_video=False,\n save_episode=False,\n)\n\nobs = env.reset()\ndone = False\nwhile not done:\n action = env.action_space.sample()\n obs, reward, done, info = env.step(action)\n```\n\n[gym]: https://github.com/openai/gym\n\n## Evaluation\n\nAgents are allowed a budget of 1M environmnent steps and are evaluated by their\nsuccess rates of the 22 achievements and by their geometric mean score. Example\nscripts for computing these are included in the `analysis` directory of the\nrepository.\n\n- **Reward:** The sparse reward is `+1` for unlocking an achievement during\n the episode and `-0.1` or `+0.1` for lost or regenerated health points.\n Results should be reported not as reward but as success rates and score.\n\n- **Success rates:** The success rates of the 22 achievemnts are computed\n as the percentage across all training episodes in which the achievement was\n unlocked, allowing insights into the ability spectrum of an agent.\n\n- **Crafter score:** The score is the geometric mean of success rates, so that\n improvements on difficult achievements contribute more than improvements on\n achievements with already high success rates.\n\n## Scoreboards\n\nPlease create a pull request if you would like to add your or another algorithm\nto the scoreboards. For the reinforcement learning and unsupervised agents\ncategories, the interaction budget is 1M. The external knowledge category is\ndefined more broadly.\n\n### Reinforcement Learning\n\n| Algorithm | Score (%) | Reward | Open Source |\n|:----------|----------:|-------:|:-----------:|\n| [Curious Replay](https://arxiv.org/pdf/2306.15934.pdf) | 19.4\u00b11.6 | - | [AutonomousAgentsLab/cr-dv3](https://github.com/AutonomousAgentsLab/cr-dv3) |\n| [PPO (ResNet)](https://arxiv.org/pdf/2307.03486.pdf)| 15.6\u00b11.6 | 10.3\u00b10.5 | [snu-mllab/Achievement-Distillation](https://github.com/snu-mllab/Achievement-Distillation) \n| [DreamerV3](https://arxiv.org/pdf/2301.04104v1.pdf) | 14.5\u00b11.6 | 11.7\u00b11.9 | [danijar/dreamerv3](https://github.com/danijar/dreamerv3) |\n| [LSTM-SPCNN](https://arxiv.org/pdf/2208.03374.pdf) | 12.1\u00b10.8 | \u2014 | [astanic/crafter-ood](https://github.com/astanic/crafter-ood) |\n| [EDE](https://openreview.net/pdf?id=GZDsKahGY-2) | 11.7\u00b11.0 | \u2014 | [yidingjiang/ede](https://github.com/yidingjiang/ede) |\n| [OC-SA](https://arxiv.org/pdf/2208.03374.pdf) | 11.1\u00b10.7 | \u2014 | [astanic/crafter-ood](https://github.com/astanic/crafter-ood) |\n| [DreamerV2](https://arxiv.org/pdf/2010.02193.pdf) | 10.0\u00b11.2 | 9.0\u00b11.7 | [danijar/dreamerv2](https://github.com/danijar/dreamerv2) |\n| [PPO](https://arxiv.org/pdf/1710.02298.pdf) | 4.6\u00b10.3 | 4.2\u00b11.2 | [DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3) |\n| [Rainbow](https://arxiv.org/pdf/1710.02298.pdf) | 4.3\u00b10.2 | 6.0\u00b11.3 | [Kaixhin/Rainbow](https://github.com/Kaixhin/Rainbow) |\n\n### Unsupervised Agents\n\n| Algorithm | Score (%) | Reward | Open Source |\n|:----------|----------:|-------:|:-----------:|\n| [Plan2Explore](https://arxiv.org/pdf/2010.02193.pdf) | 2.1\u00b10.1 | 2.1\u00b11.5 | [danijar/dreamerv2](https://github.com/danijar/dreamerv2) |\n| [RND](https://arxiv.org/pdf/1810.12894.pdf) | 2.0\u00b10.1 | 0.7\u00b11.3 | [alirezakazemipour/PPO-RND](https://github.com/alirezakazemipour/PPO-RND) |\n| Random | 1.6\u00b10.0 | 2.1\u00b11.3 | \u2014 |\n\n### External Knowledge\n\n| Algorithm | Score (%) | Reward | Uses | Interaction | Open Source |\n|:----------|----------:|-------:|:-----|:-----------:|:-----------:|\n| [Human](https://en.wikipedia.org/wiki/Human) | 50.5\u00b16.8 | 14.3\u00b12.3 | Life experience | 0 | [crafter_human_dataset](https://archive.org/details/crafter_human_dataset) |\n| [SPRING](https://arxiv.org/pdf/2305.15486.pdf) | 27.3\u00b11.2 | 12.3\u00b10.7 | LLM, scene description, Crafter paper | 0 | \u274c |\n| [Achievement Distillation](https://arxiv.org/pdf/2307.03486.pdf) | 21.8\u00b11.4 | 12.6\u00b10.3 | Reward structure | 1M | [snu-mllab/Achievement-Distillation](https://github.com/snu-mllab/Achievement-Distillation) |\n| [ELLM](https://arxiv.org/pdf/2302.06692.pdf) | \u2014 | 6.0\u00b10.4 | LLM, scene description | 5M | \u274c |\n\n## Baselines\n\nBaseline scores of various agents are available for Crafter, both with and\nwithout rewards. The scores are available in JSON format in the `scores`\ndirectory of the repository. For comparison, the score of human expert players\nis 50.5\\%. The [baseline\nimplementations](https://github.com/danijar/crafter-baselines) are available as\na separate repository.\n\n<img src=\"https://github.com/danijar/crafter/raw/main/media/scores.png\" width=\"400\"/>\n\n## Questions\n\nPlease [open an issue][issues] on Github.\n\n[issues]: https://github.com/danijar/crafter/issues\n",
"bugtrack_url": null,
"license": "",
"summary": "Open world survival game for reinforcement learning.",
"version": "1.8.2",
"project_urls": {
"Homepage": "http://github.com/danijar/crafter"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2d2a39dcc19284758fd8502ec2dad4cbde25a32cf51d33c72d76efa9cb081fa4",
"md5": "ae7e572b07c27a4f56e17d045ca58460",
"sha256": "4a142c291aa0b137c0808890381b38803876a5811b362801b8cc669faa350def"
},
"downloads": -1,
"filename": "crafter-1.8.2.tar.gz",
"has_sig": false,
"md5_digest": "ae7e572b07c27a4f56e17d045ca58460",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 107723,
"upload_time": "2023-10-31T00:46:32",
"upload_time_iso_8601": "2023-10-31T00:46:32.620644Z",
"url": "https://files.pythonhosted.org/packages/2d/2a/39dcc19284758fd8502ec2dad4cbde25a32cf51d33c72d76efa9cb081fa4/crafter-1.8.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-31 00:46:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "danijar",
"github_project": "crafter",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "crafter"
}
