# TextWorldExpress
TextWorldExpress is a highly optimized reimplementation of three text game benchmarks focusing on instruction following, commonsense reasoning, and object identification. TextWorldExpress is intended for natural language processing research, and a system description can be found in the Arxiv preprint [TextWorldExpress: Simulating Text Games at One Million Steps Per Second (PDF)](https://arxiv.org/abs/2208.01174).
#### Watch the system demonstration video on Youtube
[](https://youtu.be/MG6Ac4Xo6Ds)
# Quickstart
**Before running:** You will have to have `Java 1.8+` installed on your system (shipped with most linux distributions).
Install with pip:
conda create --name textworld-express python=3.8
conda activate textworld-express
pip install textworld-express
Run an example random agent, on the Coin Collector game...:
python examples/random_agent.py --game-name=coin
Run a user console where you can interact with the environment, on CookingWorld...:
python examples/human.py --game-name=cookingworld
# Web Server Demo
A web server demo is also available, that allows running a TextWorldExpress user console that can be interacted with in a web browser.
To run the web server demo:
conda create --name textworld-express python=3.8
conda activate textworld-express
pip install textworld-express[webserver]
Run the web server:
python examples/textworldexpress-web-server.py
Point your web browser to [localhost:8080](https://localhost:8080).
# TextWorldExpress Design
TextWorldExpress is written in Scala (2.12.9), and compiles using `sbt` into a JAR file that is run with `java`. For convenience, a `python` API is provided, which interfaces using the `py4j` package.
## Environments
TextWorldExpress includes high-speed versions of three popular benchmark environments for text-game research.
### CookingWorld ("cookingworld")
The CookingWorld environment tasks agents with preparing meals by following the instructions in a recipe that is provided in the environment. Agents must first collect required food ingredients (e.g., milk, bell pepper, flour, salt) that can be found in the environment in canonical locations (e.g., kitchen, pantry, supermarket, garden) and containers (e.g., fridge, cupboard). Randomly generated recipes require agents to first use a knife to prepare food by *slicing*, *dicing*, or *chopping* a subset of ingredients, then additionally using an appropriate heating appliance to *fry*, *roast*, or *grill* the ingredients. If all ingredients are prepared according to the recipe, the agent can use an action to *prepare the meal*, and finally *eat the meal* to complete the task successfully. Task complexity can be controlled by varying the number of locations in the environment, the number of ingredients required for the recipe, and the number of distractor ingredients randomly placed in the environment that are not required for the recipe. The recipes and environments are parametrically generated, with subsets of ingredients and specific preparations held out between training, development, and test sets to prevent overfitting. CookingWorld was originally created for the [First TextWorld Problems competition](https://aka.ms/ftwp) and later named by [[Madotto etal., 2020]](https://www.ijcai.org/proceedings/2020/207).
### TextWorld Commonsense ("twc")
Text game agents frequently learn the dynamics of environment -- such as the need to open a door before one can move through it -- from interacting with the environment itself, rather than using a pre-existing knowledge base of common sense facts or object affordances that would speed task learning. [TextWorld Commonsense](https://github.com/IBM/commonsense-rl) [[Murugesan etal., 2021]](https://arxiv.org/abs/2010.03790) aims to evaluate agents on common sense knowledge that can not be directly learned from the environment by providing agents a clean-up task where the agent must place common household objects (e.g., *a dirty dish*) in their canonical locations (e.g., *the dishwasher*) that can be found in knowledge bases such as ConceptNet. Separate lists of objects are used in the training, development, and test sets, meaning the agent can not learn object locations from the training set alone, and must rely on an external common sense knowledge base to perform well on the development and test sets. TextWorld Commonsense benchmark has three task difficulty levels, with the easiest including a single location and object to put away, while the hard setting includes up to p to 11 locations and any number of task-relevant and distractor objects.
### Coin Collector ("coin")
Agents frequently find tasks such as object search, environment navigation, or pick-and-place tasks challenging. The [Coin Collector](https://github.com/xingdi-eric-yuan/TextWorld-Coin-Collector) distills these into a single benchmark where an agent must explore a series of rooms to locate and pick up a single coin. In the original implementation, the game map typically takes the form of a connected loop or chain, such that continually moving to new locations means the agent will eventually discover the coin -- while including medium and hard modes that add in one or more "dead-end" paths. To control for environment difficulty across games, the TextWorldExpress reimplementation uses the same map generator across environments, and generates arbitrary home environments rather than connected loops. The user maintains control of other measures of difficulty, including the total number of rooms, and the number of distractor objects placed in the environment.
# Usage
### Typical Usage, and Valid Action Generation
Typical usage involves first initializing a game generator, then repeatedly generating and stepping through games. Examples are provided in the /examples/ folder, with an example agent that chooses a random action at each step described below:
```python
import random
from textworld_express import TextWorldExpressEnv
# Initialize game generator
env = TextWorldExpressEnv(envStepLimit=100)
# Set the game generator to generate a particular game (cookingworld, twc, or coin)
env.load(gameName="twc", gameParams="numLocations=3,includeDoors=1")
# Then, randomly generate and play 10 games within the defined parameters
for episode_id in range(10):
# First step
obs, infos = env.reset(seed=episode_id, gameFold="train", generateGoldPath=True)
# Display the observations from the environment.
print(obs)
for step_id in range(0, 50):
# Select a random valid action
validActions = sorted(infos['validActions'])
randomAction = random.choice(validActions)
# Take that action
obs, reward, done, infos = env.step(randomAction)
# Display action and the game's feedback.
print(">", randomAction)
print(obs)
```
### Setting Game Parameters
Environments initialize with default parameters. To change the parameters, supply a comma-delimited string into `gameParams` when calling `env.load()`. An example of a valid parameter configuration string for CookingWorld might be `numLocations=5, numIngredients=3, numDistractorItems=0, includeDoors=0, limitInventorySize=0`. Valid parameters are different for each environment, and include:
**CookingWorld:**
| Parameter | Description | Valid range |
| ----------- | ----------- | ----------- |
| numLocations | The number of locations in the environment | 1-11 |
| numIngredients | The number of ingredients to use in generating the random recipe | 1-5 |
| numDistractorItems | The number of distractor ingredients (not used in the recipe) in the environment | 0-10 |
| includeDoors | Whether rooms have doors that need to be opened | 0 or 1 |
| limitInventorySize | Whether the size of the inventory is limited | 0 or 1 |
**TextWorld Common Sense:**
| Parameter | Description | Valid range |
| ----------- | ----------- | ----------- |
| numLocations | The number of locations in the environment | 1-3 |
| numItemsToPutAway | The number of items to put away | 1-10 |
| includeDoors | Whether rooms have doors that need to be opened | 0 or 1 |
| limitInventorySize | Whether the size of the inventory is limited | 0 or 1 |
**Coin Collector:**
| Parameter | Description | Valid range |
| ----------- | ----------- | ----------- |
| numLocations | The number of locations in the environment | 1-11 |
| numDistractorItems | The number of distractor (i.e. non-coin) items in the environment | 0-10 |
| includeDoors | Whether rooms have doors that need to be opened | 0 or 1 |
| limitInventorySize | Whether the size of the inventory is limited | 0 or 1 |
**Querying current game parameters:** Sometimes you may want to know what parameters the current game is generated with. These can be queried using the `getGenerationProperties()` method:
```python
print("Generation properties: " + str(env.getGenerationProperties()))
```
### Gold paths
Gold paths can be generated by setting the `generateGoldPath` flag to be `true` when using `reset()`. The path itself can be retrieved using the `env.getGoldActionSequence()` method:
```python
print("Gold path: " + str(env.getGoldActionSequence()))
```
Note that the gold paths are generated by agents that generally perform random walks in the environment, so while they lead to successful task completion, they may not be the shortest/most efficient paths. For example, this path for Text World Common Sense wanders the environment until it either sees an object to pick up (e.g. take white coat), or an appropiate container to put an object in (e.g. put white coat in wardrobe):
> Gold path: ['look around', 'move west', 'take white coat', 'take brush', 'open wardrobe', 'put white coat in wardrobe', 'move east', 'move west', 'move east', 'move west', 'move east', 'move west', 'move east', 'move north', 'take eyeliner', 'take plaid blanket', 'put eyeliner in dressing table', 'open bathroom cabinet', 'put brush in bathroom cabinet', 'move south', 'move north', 'move south', 'move west', 'open chest of drawers', 'put plaid blanket in chest of drawers', 'move east']
### Generating Pre-crawled Paths
One of the unique features of `TextWorldExpress` is that its performance is so fast, that it becomes possible to precrawl all possible actions that a hypothetical agent might take for a given game, out to some number of steps. This has several main benefits and drawbacks:
- Positive: Game speed increases dramatically -- generally, as fast as traversing a tree using pointers. In Scala, the single-thread performance for traversal has been benchmarked at 4-5 million steps per second.
- Negative: It takes some initial time to precrawl the paths (one-time investment; generally minutes-to-hours)
- Negative: It takes time to load the paths at the start of each run (generally on the older of seconds-to-minutes)
- Negative: It can take a lot of space to store precrawled paths (generally up to 1GB per game/seed, for trees of approximately 10-12 steps, depending on the game)
- Negative: There is a limited step horizon -- once the agent goes past the number of steps precrawled, there is no more information to provide. For this reason it's important to precrawl only games that are small enough to solve within the number of steps that you precrawl.
Generally, if you are using less than a few dozen game variations during training, and you are using games that can be solved in under 10-12 steps, then path precrawling might be a way to gain dramatic benefits in performance.
To precrawl paths, use the path precrawling tool, `PathCrawler`:
java -Xmx8g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.pathcrawler.PathPrecrawler twc train 0 8 numLocations=1,includeDoors=0,numItemsToPutAway=2
(Note that you may need to supply more than `8g` of memory, depending on the size of the path being crawled.)
The arguments (in order) are the `gameName`, `gameFold`, `seed`, `maximum depth to crawl the game state tree`, and `game generation properties string`. The path crawler will export a large JSON file as output. To load these precrawled paths in Python, please check the `precrawledPathReader.py` example. For Java/Scala, please see `textworldexpress.benchmark.BenchmarkPrecrawledPath` as an end-to-end example, where `textworldexpress.pathcrawler.PrecrawledPath` provides a storage class for loading/saving precrawled paths (as well as quickly finding winning paths). Path nodes are stored internally as string-hashed storage classes (`PrecrawledNode` and `StepResultHashed`) for speed/storage efficiency, where `StepResultHashed` can be quickly converted into the normal, human-readable, unhashed version using the `StepResultHashed.toStepResult()` method. Several example precrawled paths (which are used for the benchmarking scripts) are provided in `precrawledpaths.zip`.
Path crawling can generate large files. Before path crawling, you'll likely want to make sure that the game is sized appropriately so that it can be solved within the number of steps given. Usually, this means limiting the number of locations, number of task items, etc. Below are example times and crawl sizes for a Text World Common Sense game generated with the following parameters (`numItemsToPutAway=2, numLocations=3, includeDoors=0, limitInventorySize=0`) on a 16-core machine:
| Depth | Crawl Time | Number of Nodes |
| ----------- | ----------- | ----------- |
7 | 2 sec | 198k |
8 | 23 sec | 1.6M |
9 | 209 sec | 13.5M |
# Benchmarks
### Python Benchmarks
For online generation mode:
python examples/random_agent_speed_test.py --game-name=cookingworld --max-steps=50 --num-episodes=10000
For precrawled path mode (note this demo uses precrawled paths provided for benchmarking in the repository):
unzip precrawledpaths.zip
python examples/precrawledPathReader.py
### Scala Benchmarks
For online generation mode (argument should be one of cookingworld, twc, or coin):
java -Xmx4g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.benchmark.BenchmarkOnline coin
For precrawled path mode (single-threaded, note thishis demo uses precrawled paths provided for benchmarking in the repository.
java -Xmx4g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.benchmark.BenchmarkPrecrawledPath
For extra speed, a threaded precrawled path benchmark (where here, change `32` to the desired number of threads):
java -Xmx4g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.benchmark.BenchmarkPrecrawledPathThreaded 32
# Frequently Asked Questions
**Q: Why is the Python version 10x slower than the Java/Scala version?**
A: This partially due to the `py4j` binders, that allow interfacing Python to Java/Scala code through sockets. We will look for faster solutions in the future, though the Python interface to `TextWorldExpress` is still about 100 times faster than the original TextWorld, so it is still a big win.
**Q: Will there be more benchmark games added/I want to add my own benchmark game to TextWorldExpress**
A: One of the main ways that TextWorldExpress gets its speed is through essentially hardcoding the games, mechanics, and (particularly) the valid action generation. To implement your own games, please use the existing games as templates, and open a github issue if you run into any issues. If you would like to recommend a new game to add to TextWorldExpress, please make a feature request in the github issues.
**Q: What is the fastest TextWorldExpress can run?**
A: The fastest we have clocked `TextWorldExpress` using the random agent benchmark is 4-5 million steps/sec per thread using precrawled games and the Scala native API, with multi-threaded performance at approximately 34 million steps/sec using an AMD 3950X 16-core CPU with 32 threads. This is equivalent to about 2 billion steps per minute. 2 billion steps would take a single thread of the original TextWorld about 77 days to run.
# Citation
If you use `TextWorldExpress`, please provide the following citation:
```
@article{jansen2022textworldexpress,
url = {https://arxiv.org/abs/2208.01174},
author = {Jansen, Peter A. and Côté, Marc-Alexandre},
title = {TextWorldExpress: Simulating Text Games at One Million Steps Per Second},
journal = {arXiv},
year = {2022},
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/cognitiveailab/TextWorldExpress",
"name": "textworld-express",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Peter Jansen",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/eb/3d/a35150d6ccea0ee8c858561111de4c32900f3192f190a91e1bd74722e5f6/textworld_express-1.0.3.tar.gz",
"platform": null,
"description": "# TextWorldExpress\nTextWorldExpress is a highly optimized reimplementation of three text game benchmarks focusing on instruction following, commonsense reasoning, and object identification. TextWorldExpress is intended for natural language processing research, and a system description can be found in the Arxiv preprint [TextWorldExpress: Simulating Text Games at One Million Steps Per Second (PDF)](https://arxiv.org/abs/2208.01174).\n\n#### Watch the system demonstration video on Youtube\n[](https://youtu.be/MG6Ac4Xo6Ds)\n\n\n# Quickstart\n**Before running:** You will have to have `Java 1.8+` installed on your system (shipped with most linux distributions).\n\nInstall with pip:\n\n conda create --name textworld-express python=3.8\n conda activate textworld-express\n pip install textworld-express\n\nRun an example random agent, on the Coin Collector game...:\n\n python examples/random_agent.py --game-name=coin\n\nRun a user console where you can interact with the environment, on CookingWorld...:\n\n python examples/human.py --game-name=cookingworld\n\n\n# Web Server Demo\n\nA web server demo is also available, that allows running a TextWorldExpress user console that can be interacted with in a web browser.\n\nTo run the web server demo:\n\n conda create --name textworld-express python=3.8\n conda activate textworld-express\n pip install textworld-express[webserver]\n\n\nRun the web server:\n\n python examples/textworldexpress-web-server.py\n\nPoint your web browser to [localhost:8080](https://localhost:8080).\n\n\n# TextWorldExpress Design\nTextWorldExpress is written in Scala (2.12.9), and compiles using `sbt` into a JAR file that is run with `java`. For convenience, a `python` API is provided, which interfaces using the `py4j` package.\n\n## Environments\n\nTextWorldExpress includes high-speed versions of three popular benchmark environments for text-game research.\n\n### CookingWorld (\"cookingworld\")\nThe CookingWorld environment tasks agents with preparing meals by following the instructions in a recipe that is provided in the environment. Agents must first collect required food ingredients (e.g., milk, bell pepper, flour, salt) that can be found in the environment in canonical locations (e.g., kitchen, pantry, supermarket, garden) and containers (e.g., fridge, cupboard). Randomly generated recipes require agents to first use a knife to prepare food by *slicing*, *dicing*, or *chopping* a subset of ingredients, then additionally using an appropriate heating appliance to *fry*, *roast*, or *grill* the ingredients. If all ingredients are prepared according to the recipe, the agent can use an action to *prepare the meal*, and finally *eat the meal* to complete the task successfully. Task complexity can be controlled by varying the number of locations in the environment, the number of ingredients required for the recipe, and the number of distractor ingredients randomly placed in the environment that are not required for the recipe. The recipes and environments are parametrically generated, with subsets of ingredients and specific preparations held out between training, development, and test sets to prevent overfitting. CookingWorld was originally created for the [First TextWorld Problems competition](https://aka.ms/ftwp) and later named by [[Madotto etal., 2020]](https://www.ijcai.org/proceedings/2020/207).\n\n### TextWorld Commonsense (\"twc\")\nText game agents frequently learn the dynamics of environment -- such as the need to open a door before one can move through it -- from interacting with the environment itself, rather than using a pre-existing knowledge base of common sense facts or object affordances that would speed task learning. [TextWorld Commonsense](https://github.com/IBM/commonsense-rl) [[Murugesan etal., 2021]](https://arxiv.org/abs/2010.03790) aims to evaluate agents on common sense knowledge that can not be directly learned from the environment by providing agents a clean-up task where the agent must place common household objects (e.g., *a dirty dish*) in their canonical locations (e.g., *the dishwasher*) that can be found in knowledge bases such as ConceptNet. Separate lists of objects are used in the training, development, and test sets, meaning the agent can not learn object locations from the training set alone, and must rely on an external common sense knowledge base to perform well on the development and test sets. TextWorld Commonsense benchmark has three task difficulty levels, with the easiest including a single location and object to put away, while the hard setting includes up to p to 11 locations and any number of task-relevant and distractor objects.\n\n### Coin Collector (\"coin\")\nAgents frequently find tasks such as object search, environment navigation, or pick-and-place tasks challenging. The [Coin Collector](https://github.com/xingdi-eric-yuan/TextWorld-Coin-Collector) distills these into a single benchmark where an agent must explore a series of rooms to locate and pick up a single coin. In the original implementation, the game map typically takes the form of a connected loop or chain, such that continually moving to new locations means the agent will eventually discover the coin -- while including medium and hard modes that add in one or more \"dead-end\" paths. To control for environment difficulty across games, the TextWorldExpress reimplementation uses the same map generator across environments, and generates arbitrary home environments rather than connected loops. The user maintains control of other measures of difficulty, including the total number of rooms, and the number of distractor objects placed in the environment.\n\n# Usage\n\n### Typical Usage, and Valid Action Generation\nTypical usage involves first initializing a game generator, then repeatedly generating and stepping through games. Examples are provided in the /examples/ folder, with an example agent that chooses a random action at each step described below:\n\n```python\nimport random\nfrom textworld_express import TextWorldExpressEnv\n\n# Initialize game generator\nenv = TextWorldExpressEnv(envStepLimit=100)\n\n# Set the game generator to generate a particular game (cookingworld, twc, or coin)\nenv.load(gameName=\"twc\", gameParams=\"numLocations=3,includeDoors=1\")\n\n# Then, randomly generate and play 10 games within the defined parameters\nfor episode_id in range(10):\n # First step\n obs, infos = env.reset(seed=episode_id, gameFold=\"train\", generateGoldPath=True)\n\n # Display the observations from the environment.\n print(obs)\n\n for step_id in range(0, 50):\n # Select a random valid action\n validActions = sorted(infos['validActions'])\n randomAction = random.choice(validActions)\n\n # Take that action\n obs, reward, done, infos = env.step(randomAction)\n\n # Display action and the game's feedback.\n print(\">\", randomAction)\n print(obs)\n\n```\n\n### Setting Game Parameters\nEnvironments initialize with default parameters. To change the parameters, supply a comma-delimited string into `gameParams` when calling `env.load()`. An example of a valid parameter configuration string for CookingWorld might be `numLocations=5, numIngredients=3, numDistractorItems=0, includeDoors=0, limitInventorySize=0`. Valid parameters are different for each environment, and include:\n\n**CookingWorld:**\n| Parameter | Description | Valid range |\n| ----------- | ----------- | ----------- |\n| numLocations | The number of locations in the environment | 1-11 |\n| numIngredients | The number of ingredients to use in generating the random recipe | 1-5 |\n| numDistractorItems | The number of distractor ingredients (not used in the recipe) in the environment | 0-10 |\n| includeDoors | Whether rooms have doors that need to be opened | 0 or 1 |\n| limitInventorySize | Whether the size of the inventory is limited | 0 or 1 |\n\n**TextWorld Common Sense:**\n| Parameter | Description | Valid range |\n| ----------- | ----------- | ----------- |\n| numLocations | The number of locations in the environment | 1-3 |\n| numItemsToPutAway | The number of items to put away | 1-10 |\n| includeDoors | Whether rooms have doors that need to be opened | 0 or 1 |\n| limitInventorySize | Whether the size of the inventory is limited | 0 or 1 |\n\n**Coin Collector:**\n| Parameter | Description | Valid range |\n| ----------- | ----------- | ----------- |\n| numLocations | The number of locations in the environment | 1-11 |\n| numDistractorItems | The number of distractor (i.e. non-coin) items in the environment | 0-10 |\n| includeDoors | Whether rooms have doors that need to be opened | 0 or 1 |\n| limitInventorySize | Whether the size of the inventory is limited | 0 or 1 |\n\n**Querying current game parameters:** Sometimes you may want to know what parameters the current game is generated with. These can be queried using the `getGenerationProperties()` method:\n```python\nprint(\"Generation properties: \" + str(env.getGenerationProperties()))\n```\n\n\n### Gold paths\n\nGold paths can be generated by setting the `generateGoldPath` flag to be `true` when using `reset()`. The path itself can be retrieved using the `env.getGoldActionSequence()` method:\n\n```python\nprint(\"Gold path: \" + str(env.getGoldActionSequence()))\n```\n\nNote that the gold paths are generated by agents that generally perform random walks in the environment, so while they lead to successful task completion, they may not be the shortest/most efficient paths. For example, this path for Text World Common Sense wanders the environment until it either sees an object to pick up (e.g. take white coat), or an appropiate container to put an object in (e.g. put white coat in wardrobe):\n\n> Gold path: ['look around', 'move west', 'take white coat', 'take brush', 'open wardrobe', 'put white coat in wardrobe', 'move east', 'move west', 'move east', 'move west', 'move east', 'move west', 'move east', 'move north', 'take eyeliner', 'take plaid blanket', 'put eyeliner in dressing table', 'open bathroom cabinet', 'put brush in bathroom cabinet', 'move south', 'move north', 'move south', 'move west', 'open chest of drawers', 'put plaid blanket in chest of drawers', 'move east']\n\n### Generating Pre-crawled Paths\n\nOne of the unique features of `TextWorldExpress` is that its performance is so fast, that it becomes possible to precrawl all possible actions that a hypothetical agent might take for a given game, out to some number of steps. This has several main benefits and drawbacks:\n- Positive: Game speed increases dramatically -- generally, as fast as traversing a tree using pointers. In Scala, the single-thread performance for traversal has been benchmarked at 4-5 million steps per second.\n- Negative: It takes some initial time to precrawl the paths (one-time investment; generally minutes-to-hours)\n- Negative: It takes time to load the paths at the start of each run (generally on the older of seconds-to-minutes)\n- Negative: It can take a lot of space to store precrawled paths (generally up to 1GB per game/seed, for trees of approximately 10-12 steps, depending on the game)\n- Negative: There is a limited step horizon -- once the agent goes past the number of steps precrawled, there is no more information to provide. For this reason it's important to precrawl only games that are small enough to solve within the number of steps that you precrawl.\n\nGenerally, if you are using less than a few dozen game variations during training, and you are using games that can be solved in under 10-12 steps, then path precrawling might be a way to gain dramatic benefits in performance.\n\nTo precrawl paths, use the path precrawling tool, `PathCrawler`:\n\n java -Xmx8g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.pathcrawler.PathPrecrawler twc train 0 8 numLocations=1,includeDoors=0,numItemsToPutAway=2\n\n(Note that you may need to supply more than `8g` of memory, depending on the size of the path being crawled.)\n\nThe arguments (in order) are the `gameName`, `gameFold`, `seed`, `maximum depth to crawl the game state tree`, and `game generation properties string`. The path crawler will export a large JSON file as output. To load these precrawled paths in Python, please check the `precrawledPathReader.py` example. For Java/Scala, please see `textworldexpress.benchmark.BenchmarkPrecrawledPath` as an end-to-end example, where `textworldexpress.pathcrawler.PrecrawledPath` provides a storage class for loading/saving precrawled paths (as well as quickly finding winning paths). Path nodes are stored internally as string-hashed storage classes (`PrecrawledNode` and `StepResultHashed`) for speed/storage efficiency, where `StepResultHashed` can be quickly converted into the normal, human-readable, unhashed version using the `StepResultHashed.toStepResult()` method. Several example precrawled paths (which are used for the benchmarking scripts) are provided in `precrawledpaths.zip`.\n\nPath crawling can generate large files. Before path crawling, you'll likely want to make sure that the game is sized appropriately so that it can be solved within the number of steps given. Usually, this means limiting the number of locations, number of task items, etc. Below are example times and crawl sizes for a Text World Common Sense game generated with the following parameters (`numItemsToPutAway=2, numLocations=3, includeDoors=0, limitInventorySize=0`) on a 16-core machine:\n| Depth | Crawl Time | Number of Nodes |\n| ----------- | ----------- | ----------- |\n7 | 2 sec | 198k |\n8 | 23 sec | 1.6M |\n9 | 209 sec | 13.5M |\n\n# Benchmarks\n\n### Python Benchmarks\nFor online generation mode:\n\n python examples/random_agent_speed_test.py --game-name=cookingworld --max-steps=50 --num-episodes=10000\n\nFor precrawled path mode (note this demo uses precrawled paths provided for benchmarking in the repository):\n\n unzip precrawledpaths.zip\n python examples/precrawledPathReader.py\n\n### Scala Benchmarks\nFor online generation mode (argument should be one of cookingworld, twc, or coin):\n\n java -Xmx4g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.benchmark.BenchmarkOnline coin\n\nFor precrawled path mode (single-threaded, note thishis demo uses precrawled paths provided for benchmarking in the repository.\n\n java -Xmx4g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.benchmark.BenchmarkPrecrawledPath\n\nFor extra speed, a threaded precrawled path benchmark (where here, change `32` to the desired number of threads):\n\n java -Xmx4g -cp textworld_express/textworld-express-1.0.0.jar textworldexpress.benchmark.BenchmarkPrecrawledPathThreaded 32\n\n\n# Frequently Asked Questions\n**Q: Why is the Python version 10x slower than the Java/Scala version?**\nA: This partially due to the `py4j` binders, that allow interfacing Python to Java/Scala code through sockets. We will look for faster solutions in the future, though the Python interface to `TextWorldExpress` is still about 100 times faster than the original TextWorld, so it is still a big win.\n\n**Q: Will there be more benchmark games added/I want to add my own benchmark game to TextWorldExpress**\nA: One of the main ways that TextWorldExpress gets its speed is through essentially hardcoding the games, mechanics, and (particularly) the valid action generation. To implement your own games, please use the existing games as templates, and open a github issue if you run into any issues. If you would like to recommend a new game to add to TextWorldExpress, please make a feature request in the github issues.\n\n**Q: What is the fastest TextWorldExpress can run?**\nA: The fastest we have clocked `TextWorldExpress` using the random agent benchmark is 4-5 million steps/sec per thread using precrawled games and the Scala native API, with multi-threaded performance at approximately 34 million steps/sec using an AMD 3950X 16-core CPU with 32 threads. This is equivalent to about 2 billion steps per minute. 2 billion steps would take a single thread of the original TextWorld about 77 days to run.\n\n\n# Citation\nIf you use `TextWorldExpress`, please provide the following citation:\n```\n@article{jansen2022textworldexpress,\n url = {https://arxiv.org/abs/2208.01174},\n author = {Jansen, Peter A. and C\u00f4t\u00e9, Marc-Alexandre},\n title = {TextWorldExpress: Simulating Text Games at One Million Steps Per Second},\n journal = {arXiv},\n year = {2022},\n}\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "TextWorldExpress: a highly optimized reimplementation of three text game benchmarks focusing on instruction following, commonsense reasoning, and object identification.",
"version": "1.0.3",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "b4d6df1fe98134c02f39554330580e07",
"sha256": "22e646a7345fd6ec0e7b24c89eba749c4e92a6866b59e38c5b1c22f0c9599aad"
},
"downloads": -1,
"filename": "textworld_express-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b4d6df1fe98134c02f39554330580e07",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 1495490,
"upload_time": "2022-12-02T03:05:25",
"upload_time_iso_8601": "2022-12-02T03:05:25.614041Z",
"url": "https://files.pythonhosted.org/packages/82/29/656720ef30220585d779e514cb3d9d5a0a891d95a734ec8a1b760d0d5bc8/textworld_express-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "2d084a072d5e04fd92dc7bbead7a5e2e",
"sha256": "629687e04034bf4bd81e4b93eb46d25670addb2b2d15b20cd47bb40a43250f32"
},
"downloads": -1,
"filename": "textworld_express-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "2d084a072d5e04fd92dc7bbead7a5e2e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1504036,
"upload_time": "2022-12-02T03:05:28",
"upload_time_iso_8601": "2022-12-02T03:05:28.347731Z",
"url": "https://files.pythonhosted.org/packages/eb/3d/a35150d6ccea0ee8c858561111de4c32900f3192f190a91e1bd74722e5f6/textworld_express-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-02 03:05:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "cognitiveailab",
"github_project": "TextWorldExpress",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "py4j",
"specs": []
},
{
"name": "orjson",
"specs": []
},
{
"name": "prompt_toolkit",
"specs": []
}
],
"lcname": "textworld-express"
}
JSON
