# Duplicate Image Finder (difPy)
**Tired of going through all images in a folder and comparing them manually to check if they are duplicates?**
The Duplicate Image Finder (difPy) Python package **automates** this task for you!
Read more on how the algorithm of difPy works in my Medium article **[Finding Duplicate Images with Python](https://towardsdatascience.com/finding-duplicate-images-with-python-71c04ec8051)**.
For a **detailed usage guide**, please view the official **[difPy Usage Documentation](https://difpy.readthedocs.io/)**.
-------
## Description
difPy searches for images in **one or more different folders**, compares the images it found and checks whether these are duplicates. It then outputs the **image files classified as duplicates** as well as the **images having the lowest resolutions**, so you know which of the duplicate images are safe to be deleted. You can then either delete them manually, or let difPy delete them for you.
difPy does not compare images based on their hashes. It compares them based on their tensors i. e. the image content - this allows difPy to **not only search for duplicate images, but also for similar images**.
difPy leverages Python's **multiprocessing capabilities** and is therefore able to perform at high performance even on large datasets.
## Table of Contents
1. [Basic Usage](https://github.com/elisemercury/Duplicate-Image-Finder#basic-usage)
2. [Output](https://github.com/elisemercury/Duplicate-Image-Finder#output)
3. [Additional Parameters](https://github.com/elisemercury/Duplicate-Image-Finder#additional-parameters)
4. [CLI Usage](https://github.com/elisemercury/Duplicate-Image-Finder#cli-usage)
## Basic Usage
To make difPy search for duplicates **within one folder**:
```python
import difPy
dif = difPy.build('C:/Path/to/Folder/')
search = difPy.search(dif)
```
To search for duplicates **within multiple folders**:
```python
import difPy
dif = difPy.build(['C:/Path/to/FolderA/', 'C:/Path/to/FolderB/', 'C:/Path/to/FolderC/',...])
search = difPy.search(dif)
```
Folder paths can be specified as standalone Python strings, or within a list. With `difPy.build()`, difPy first scans the images in the provided folders and builds a collection of images by generating image tensors. `difPy.search()` then starts the search for duplicate images.
## Output
difPy returns various types of output that you may use depending on your use case:
### I. Search Result
A **JSON formatted collection** of duplicates/similar images (i. e. **match groups**) that were found. Each match group has a primary image (the key of the dictionary) which holds the list of its duplicates including their filename and MSE (Mean Squared Error). The lower the MSE, the more similar the primary image and the matched images are. Therefore, an MSE of 0 indicates that two images are exact duplicates.
```python
search.result
> Output:
{'C:/Path/to/Image/image1.jpg' : [['C:/Path/to/Image/duplicate_image1a.jpg', 0.0],
['C:/Path/to/Image/duplicate_image1b.jpg', 0.0]],
'C:/Path/to/Image/image2.jpg' : [['C:/Path/to/Image/duplicate_image2a.jpg', 0.0]],
...
}
```
### II. Lower Quality Files
A **list** of duplicates/similar images that have the **lowest quality** (image resolution) among match groups:
```python
search.lower_quality
> Output:
['C:/Path/to/Image/duplicate_image1.jpg',
'C:/Path/to/Image/duplicate_image2.jpg', ...]
```
Lower quality images then can be **moved to a different location**:
```python
search.move_to(destination_path='C:/Path/to/Destination/')
```
Or **deleted**:
```python
search.delete(silent_del=False)
```
### III. Search Statistics
A **JSON formatted collection** with statistics on the completed difPy processes:
```python
search.stats
> Output:
{'directory': ['C:/Path/to/Folder_A/', 'C:/Path/to/Folder_B/', ... ],
'process': {'build': {'duration': {'start': '2024-02-18T19:52:39.479548',
'end': '2024-02-18T19:52:41.630027',
'seconds_elapsed': 2.1505},
'parameters': {'recursive': True,
'in_folder': False,
'limit_extensions': True,
'px_size': 50,
'processes': 5}},
'search': {'duration': {'start': '2024-02-18T19:52:41.630027',
'end': '2024-02-18T19:52:46.770077',
'seconds_elapsed': 5.14},
'parameters': {'similarity_mse': 0,
'rotate': True,
'same_dim': True,
'processes': 5,
'chunksize': None},
'files_searched': 3232,
'matches_found': {'duplicates': 3030,
'similar': 0}}},
'total_files': {'count': 3232},
'invalid_files': {'count': 0,
'logs': {}}}
```
## Additional Parameters
difPy supports the following parameters:
```python
difPy.build(*directory, recursive=True, in_folder=False, limit_extensions=True,
px_size=50, show_progress=True, processes=os.cpu_count())
```
```python
difPy.search(difpy_obj, similarity='duplicates', rotate=True, same_dim=True,
show_progress=True, processes=os.cpu_count(), chunksize=None)
```
## CLI Usage
difPy can also be invoked through the CLI by using the following commands:
```python
python dif.py #working directory
python dif.py -D 'C:/Path/to/Folder/'
python dif.py -D 'C:/Path/to/Folder_A/' 'C:/Path/to/Folder_B/' 'C:/Path/to/Folder_C/'
```
difPy CLI supports the following arguments:
```python
dif.py [-h] [-D DIRECTORY [DIRECTORY ...]] [-Z OUTPUT_DIRECTORY]
[-r {True,False}] [-i {True,False}] [-le {True,False}]
[-px PX_SIZE] [-s SIMILARITY] [-ro {True,False}]
[-dim {True,False}] [-proc PROCESSES] [-ch CHUNKSIZE]
[-mv MOVE_TO] [-d {True,False}] [-sd {True,False}]
[-p {True,False}]
```
If no directory parameter is given in the CLI, difPy will **run on the current working directory**.
When running from the CLI, the output of difPy is written to files and **saved in the working directory** by default. To change the default output directory, specify the `-Z / -output_directory` parameter. The "xxx" in the output filenames is the current timestamp:
```python
difPy_xxx_results.json
difPy_xxx_lower_quality.json
difPy_xxx_stats.json
```
-------
For a **detailed usage guide**, please view the official **[difPy Usage Documentation](https://difpy.readthedocs.io/)**.
Raw data
{
"_id": null,
"home_page": "https://github.com/elisemercury/Duplicate-Image-Finder",
"name": "difPy",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "duplicate, image, finder, similarity, pictures",
"author": "Elise Landman",
"author_email": "elisejlandman@hotmail.com",
"download_url": "https://files.pythonhosted.org/packages/b5/97/37b0c884a1736fb39721bafb7b83fb74471a53a40fab268cc2a69a9da642/difPy-4.2.0.tar.gz",
"platform": null,
"description": "# Duplicate Image Finder (difPy)\r\n\r\n**Tired of going through all images in a folder and comparing them manually to check if they are duplicates?**\r\n\r\nThe Duplicate Image Finder (difPy) Python package **automates** this task for you!\r\n\r\nRead more on how the algorithm of difPy works in my Medium article **[Finding Duplicate Images with Python](https://towardsdatascience.com/finding-duplicate-images-with-python-71c04ec8051)**.\r\n\r\nFor a **detailed usage guide**, please view the official **[difPy Usage Documentation](https://difpy.readthedocs.io/)**.\r\n\r\n-------\r\n\r\n## Description\r\ndifPy searches for images in **one or more different folders**, compares the images it found and checks whether these are duplicates. It then outputs the **image files classified as duplicates** as well as the **images having the lowest resolutions**, so you know which of the duplicate images are safe to be deleted. You can then either delete them manually, or let difPy delete them for you.\r\n\r\ndifPy does not compare images based on their hashes. It compares them based on their tensors i. e. the image content - this allows difPy to **not only search for duplicate images, but also for similar images**.\r\n\r\ndifPy leverages Python's **multiprocessing capabilities** and is therefore able to perform at high performance even on large datasets. \r\n\r\n\r\n## Table of Contents\r\n1. [Basic Usage](https://github.com/elisemercury/Duplicate-Image-Finder#basic-usage)\r\n2. [Output](https://github.com/elisemercury/Duplicate-Image-Finder#output)\r\n3. [Additional Parameters](https://github.com/elisemercury/Duplicate-Image-Finder#additional-parameters)\r\n4. [CLI Usage](https://github.com/elisemercury/Duplicate-Image-Finder#cli-usage)\r\n\r\n## Basic Usage\r\nTo make difPy search for duplicates **within one folder**:\r\n\r\n```python\r\nimport difPy\r\ndif = difPy.build('C:/Path/to/Folder/')\r\nsearch = difPy.search(dif)\r\n``` \r\nTo search for duplicates **within multiple folders**:\r\n\r\n```python\r\nimport difPy\r\ndif = difPy.build(['C:/Path/to/FolderA/', 'C:/Path/to/FolderB/', 'C:/Path/to/FolderC/',...])\r\nsearch = difPy.search(dif)\r\n``` \r\n\r\nFolder paths can be specified as standalone Python strings, or within a list. With `difPy.build()`, difPy first scans the images in the provided folders and builds a collection of images by generating image tensors. `difPy.search()` then starts the search for duplicate images.\r\n\r\n## Output\r\ndifPy returns various types of output that you may use depending on your use case: \r\n\r\n### I. Search Result\r\nA **JSON formatted collection** of duplicates/similar images (i. e. **match groups**) that were found. Each match group has a primary image (the key of the dictionary) which holds the list of its duplicates including their filename and MSE (Mean Squared Error). The lower the MSE, the more similar the primary image and the matched images are. Therefore, an MSE of 0 indicates that two images are exact duplicates.\r\n\r\n```python\r\nsearch.result\r\n\r\n> Output:\r\n{'C:/Path/to/Image/image1.jpg' : [['C:/Path/to/Image/duplicate_image1a.jpg', 0.0], \r\n ['C:/Path/to/Image/duplicate_image1b.jpg', 0.0]],\r\n 'C:/Path/to/Image/image2.jpg' : [['C:/Path/to/Image/duplicate_image2a.jpg', 0.0]],\r\n ...\r\n}\r\n``` \r\n\r\n### II. Lower Quality Files\r\nA **list** of duplicates/similar images that have the **lowest quality** (image resolution) among match groups: \r\n\r\n```python\r\nsearch.lower_quality\r\n\r\n> Output:\r\n['C:/Path/to/Image/duplicate_image1.jpg', \r\n 'C:/Path/to/Image/duplicate_image2.jpg', ...]\r\n``` \r\n\r\nLower quality images then can be **moved to a different location**:\r\n\r\n```python\r\nsearch.move_to(destination_path='C:/Path/to/Destination/')\r\n```\r\nOr **deleted**:\r\n\r\n```python\r\nsearch.delete(silent_del=False)\r\n```\r\n\r\n### III. Search Statistics\r\n\r\nA **JSON formatted collection** with statistics on the completed difPy processes:\r\n\r\n```python\r\nsearch.stats\r\n\r\n> Output:\r\n{'directory': ['C:/Path/to/Folder_A/', 'C:/Path/to/Folder_B/', ... ],\r\n 'process': {'build': {'duration': {'start': '2024-02-18T19:52:39.479548',\r\n 'end': '2024-02-18T19:52:41.630027',\r\n 'seconds_elapsed': 2.1505},\r\n 'parameters': {'recursive': True,\r\n 'in_folder': False,\r\n 'limit_extensions': True,\r\n 'px_size': 50,\r\n 'processes': 5}},\r\n 'search': {'duration': {'start': '2024-02-18T19:52:41.630027',\r\n 'end': '2024-02-18T19:52:46.770077',\r\n 'seconds_elapsed': 5.14},\r\n 'parameters': {'similarity_mse': 0,\r\n 'rotate': True,\r\n 'same_dim': True,\r\n 'processes': 5,\r\n 'chunksize': None},\r\n 'files_searched': 3232,\r\n 'matches_found': {'duplicates': 3030, \r\n 'similar': 0}}},\r\n 'total_files': {'count': 3232},\r\n 'invalid_files': {'count': 0, \r\n 'logs': {}}}\r\n```\r\n\r\n## Additional Parameters\r\ndifPy supports the following parameters:\r\n\r\n```python\r\ndifPy.build(*directory, recursive=True, in_folder=False, limit_extensions=True, \r\n px_size=50, show_progress=True, processes=os.cpu_count())\r\n```\r\n\r\n```python\r\ndifPy.search(difpy_obj, similarity='duplicates', rotate=True, same_dim=True, \r\n show_progress=True, processes=os.cpu_count(), chunksize=None)\r\n```\r\n\r\n## CLI Usage\r\ndifPy can also be invoked through the CLI by using the following commands:\r\n\r\n```python\r\npython dif.py #working directory\r\n\r\npython dif.py -D 'C:/Path/to/Folder/'\r\n\r\npython dif.py -D 'C:/Path/to/Folder_A/' 'C:/Path/to/Folder_B/' 'C:/Path/to/Folder_C/'\r\n```\r\n\r\ndifPy CLI supports the following arguments:\r\n\r\n```python\r\ndif.py [-h] [-D DIRECTORY [DIRECTORY ...]] [-Z OUTPUT_DIRECTORY] \r\n [-r {True,False}] [-i {True,False}] [-le {True,False}] \r\n [-px PX_SIZE] [-s SIMILARITY] [-ro {True,False}]\r\n [-dim {True,False}] [-proc PROCESSES] [-ch CHUNKSIZE] \r\n [-mv MOVE_TO] [-d {True,False}] [-sd {True,False}]\r\n [-p {True,False}]\r\n```\r\n\r\nIf no directory parameter is given in the CLI, difPy will **run on the current working directory**.\r\n\r\nWhen running from the CLI, the output of difPy is written to files and **saved in the working directory** by default. To change the default output directory, specify the `-Z / -output_directory` parameter. The \"xxx\" in the output filenames is the current timestamp:\r\n\r\n```python\r\ndifPy_xxx_results.json\r\ndifPy_xxx_lower_quality.json\r\ndifPy_xxx_stats.json\r\n```\r\n\r\n-------\r\n\r\nFor a **detailed usage guide**, please view the official **[difPy Usage Documentation](https://difpy.readthedocs.io/)**.\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "difPy Duplicate Image Finder - automated search for duplicate or similar images.",
"version": "4.2.0",
"project_urls": {
"Download": "https://github.com/elisemercury/Duplicate-Image-Finder/archive/refs/tags/v4.2.0.tar.gz",
"Homepage": "https://github.com/elisemercury/Duplicate-Image-Finder"
},
"split_keywords": [
"duplicate",
" image",
" finder",
" similarity",
" pictures"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "31d9fd5a16694d95448edd30bcb3d9449835156a9fe38c442278c346b6a6b735",
"md5": "fdc8aea13ae709c9d2ff378da4e60582",
"sha256": "0ea46ba55a8eb46d6c8bc252b36b7ecd6c6bea45e715a57131ff886943b3706c"
},
"downloads": -1,
"filename": "difPy-4.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fdc8aea13ae709c9d2ff378da4e60582",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 18821,
"upload_time": "2025-01-07T22:35:24",
"upload_time_iso_8601": "2025-01-07T22:35:24.525452Z",
"url": "https://files.pythonhosted.org/packages/31/d9/fd5a16694d95448edd30bcb3d9449835156a9fe38c442278c346b6a6b735/difPy-4.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b59737b0c884a1736fb39721bafb7b83fb74471a53a40fab268cc2a69a9da642",
"md5": "191089505107b41e66209541388e5608",
"sha256": "07925f757ca49ea06c5cbb58ce45c5d5a24e63e72a7d26eb8ba48d272e105a5f"
},
"downloads": -1,
"filename": "difPy-4.2.0.tar.gz",
"has_sig": false,
"md5_digest": "191089505107b41e66209541388e5608",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 21744,
"upload_time": "2025-01-07T22:35:27",
"upload_time_iso_8601": "2025-01-07T22:35:27.507560Z",
"url": "https://files.pythonhosted.org/packages/b5/97/37b0c884a1736fb39721bafb7b83fb74471a53a40fab268cc2a69a9da642/difPy-4.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-07 22:35:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "elisemercury",
"github_project": "Duplicate-Image-Finder",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "difpy"
}
JSON
