# ShapeY
ShapeY is a benchmark that tests a vision system's shape recognition capacity. ShapeY currently consists of ~68k images of 200 3D objects taken from ShapeNet. Note that this benchmark is not meant to be used as a training dataset, but rather serves to validate that the visual object recogntion / classification under inspection has developed a capacity to perform well on our benchmarking tasks, which are designed to be hard if the system does not understand shape.
## Installing ShapeY
Requirements: Python 3, Cuda version 10.2 (prerequisite for cupy)
To install ShapeY, run the following command:
```
pip3 install shapey==0.1.7
```
## Step0: Download ShapeY200 dataset
Run `download.sh` to download the dataset. The script automatically unzips the images under `data/ShapeY200/`.
Downloading uses gdown, which is google drive command line tool. If it does not work, please just follow the two links down below to download the ShapeY200 / ShapeY200CR datasets.
ShapeY200:
https://drive.google.com/uc?id=1arDu0c9hYLHVMiB52j_a-e0gVnyQfuQV
ShapeY200CR:
https://drive.google.com/uc?id=1WXpNUVRn6D0F9T3IHruml2DcDCFRsix-
After downloading the two datasets, move each of them to the `data/` directory. For example, all of the images for ShapeY200 should be under `data/ShapeY200/dataset/`.
## Step1: Extract the embedding vectors from your own vision model using our dataset
Implement the function `your_feature_output_code` in `step1_save_feature/your_feature_extraction_code.py`. The function should take in the path to the dataset as input and return two lists - one for the image names and another for the corresponding embedding vectors taken from whatever system.
## Step2: Run macro.py
After you have implemented the function, run `macro.py` to generate the results.
`macro.py` will automatically run the following steps:
1. Compute correlation between all embedding vectors (using `step2_compute_feature_correlation/compute_correlation.py`)
2. Run benchmark analysis (using `step3_benchmark_analysis/get_nn_classification_error_with_exclusion_distance.py`)
3. Graph results (top1 object matching error, top1 category matching error, etc.)
Raw data
{
"_id": null,
"home_page": "https://github.com/njw0709/ShapeY",
"name": "ShapeY",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "tests shape recognition capacity",
"author": "Jong Woo Nam",
"author_email": "namj@usc.edu",
"download_url": "https://files.pythonhosted.org/packages/3d/83/3a9496be1f7487ffec581d96d96f49b4dc20b5aa462d3aadb158da579008/ShapeY-0.1.8.tar.gz",
"platform": null,
"description": "# ShapeY\n\nShapeY is a benchmark that tests a vision system's shape recognition capacity. ShapeY currently consists of ~68k images of 200 3D objects taken from ShapeNet. Note that this benchmark is not meant to be used as a training dataset, but rather serves to validate that the visual object recogntion / classification under inspection has developed a capacity to perform well on our benchmarking tasks, which are designed to be hard if the system does not understand shape.\n\n## Installing ShapeY\nRequirements: Python 3, Cuda version 10.2 (prerequisite for cupy)\n\nTo install ShapeY, run the following command:\n```\npip3 install shapey==0.1.7\n```\n\n## Step0: Download ShapeY200 dataset\nRun `download.sh` to download the dataset. The script automatically unzips the images under `data/ShapeY200/`.\nDownloading uses gdown, which is google drive command line tool. If it does not work, please just follow the two links down below to download the ShapeY200 / ShapeY200CR datasets.\n\nShapeY200:\nhttps://drive.google.com/uc?id=1arDu0c9hYLHVMiB52j_a-e0gVnyQfuQV\n\nShapeY200CR:\nhttps://drive.google.com/uc?id=1WXpNUVRn6D0F9T3IHruml2DcDCFRsix-\n\nAfter downloading the two datasets, move each of them to the `data/` directory. For example, all of the images for ShapeY200 should be under `data/ShapeY200/dataset/`.\n\n## Step1: Extract the embedding vectors from your own vision model using our dataset\nImplement the function `your_feature_output_code` in `step1_save_feature/your_feature_extraction_code.py`. The function should take in the path to the dataset as input and return two lists - one for the image names and another for the corresponding embedding vectors taken from whatever system.\n\n## Step2: Run macro.py\nAfter you have implemented the function, run `macro.py` to generate the results.\n`macro.py` will automatically run the following steps:\n1. Compute correlation between all embedding vectors (using `step2_compute_feature_correlation/compute_correlation.py`)\n\n2. Run benchmark analysis (using `step3_benchmark_analysis/get_nn_classification_error_with_exclusion_distance.py`)\n\n3. Graph results (top1 object matching error, top1 category matching error, etc.)",
"bugtrack_url": null,
"license": "MIT",
"summary": "Benchmark that tests shape recognition",
"version": "0.1.8",
"split_keywords": [
"tests",
"shape",
"recognition",
"capacity"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "80204e94c3fb61d40aca29e663b39497",
"sha256": "7386d460965c89f88ff6a425db145d1a40d0e045d12072c3330a7ad9c293e33b"
},
"downloads": -1,
"filename": "ShapeY-0.1.8.tar.gz",
"has_sig": false,
"md5_digest": "80204e94c3fb61d40aca29e663b39497",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 19857,
"upload_time": "2022-12-21T22:49:24",
"upload_time_iso_8601": "2022-12-21T22:49:24.094475Z",
"url": "https://files.pythonhosted.org/packages/3d/83/3a9496be1f7487ffec581d96d96f49b4dc20b5aa462d3aadb158da579008/ShapeY-0.1.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-21 22:49:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "njw0709",
"github_project": "ShapeY",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "torch",
"specs": [
[
">=",
"1.10.0"
]
]
},
{
"name": "torchvision",
"specs": [
[
">=",
"0.11.2"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.18.4"
]
]
},
{
"name": "kaleido",
"specs": [
[
">=",
"0.0.1"
]
]
},
{
"name": "tqdm",
"specs": [
[
">=",
"4.48.0"
]
]
},
{
"name": "cupy-cuda102",
"specs": []
},
{
"name": "matplotlib",
"specs": [
[
">=",
"3.3.1"
]
]
},
{
"name": "gdown",
"specs": []
},
{
"name": "h5py",
"specs": [
[
">=",
"2.10.0"
]
]
},
{
"name": "hydra",
"specs": [
[
">=",
"1.1.1"
]
]
},
{
"name": "twine",
"specs": []
},
{
"name": "black",
"specs": []
},
{
"name": "pylint",
"specs": []
}
],
"lcname": "shapey"
}
JSON
