%2520-yellow%3Fstyle%3Dfor-the-badge)
A powerful open-source tool for analyzing image and video datasets founded by the authors of XGBoost, Apache TVM & Turi Create - Danny Bickson, Carlos Guestrin and Amir Alush.
Documentation
·
Features
·
Report Bug
·
Blog
·
Quickstart
·
Visual Layer Cloud
Getting Started
pip install fastdup from PyPI:
pip install fastdup
More installation options are available here.
Initialize and run fastdup:
import fastdup
fd = fastdup.create(input_dir="IMAGE_FOLDER/")
fd.run()
Remove duplicates from your dataset in a single call:
import fastdup
fastdup.remove_duplicates("IMAGE_FOLDER/")
This finds and deletes duplicate images (similarity > 0.96 by default) directly from disk. Use dry_run=True to preview which files would be removed before deleting, and adjust the distance parameter to control the similarity threshold.
Visualize the results in a static gallery:
fd.vis.duplicates_gallery() # gallery of duplicates
fd.vis.outliers_gallery() # gallery of outliers
fd.vis.component_gallery() # gallery of connected components
fd.vis.stats_gallery() # gallery of image statistics (e.g. blur, brightness, etc.)
fd.vis.similarity_gallery() # gallery of similar images
https://github.com/user-attachments/assets/738a329d-8063-4515-a961-f2527934a0ca
Features & Advantages
fastdup handles labeled/unlabeled datasets in image or video format, providing a range of features:
What sets fastdup apart from other similar tools:
- Quality: High-quality analysis to identify duplicates/near-duplicates, outliers, mislabels, broken images, and low-quality images.
- Scale: Highly scalable, capable of processing 400M images on a single CPU machine. Scales up to billions of images.
- Speed: Optimized C++ engine enables high performance even on low-resource CPU machines.
- Privacy: Runs locally or on your cloud infrastructure. Your data stays where it is.
- Ease of use: Works on labeled or unlabeled datasets in image or video format with support for major operating systems like MacOS, Linux and Windows.
Learn from Examples
Learn the basics of fastdup through interactive examples. View the notebooks on GitHub or nbviewer. Even better, run them on Google Colab or Kaggle, for free.
|
⚡ Quickstart: Learn how to install fastdup, load a dataset and analyze it for potential issues such as duplicates/near-duplicates, broken images, outliers, dark/bright/blurry images, and view visually similar image clusters. If you're new, start here!
📌 Dataset: Oxford-IIIT Pet.
|
|
|
|
|
|
🧹 Finding and Removing Duplicates: Learn how to how to analyze an image dataset for duplicates and near-duplicates.
📌 Dataset: Oxford-IIIT Pet.
|
|
|
|
|
|
|
|
|
🖼 Finding and Removing Mislabels: Learn how to analyze an image dataset for potential image mislabels and export the list of mislabeled images for further inspection.
📌 Dataset: Food-101.
|
|
|
|
|
|
|
|
|
🎁 Image Similarity Search: Perform image search in a large dataset of images.
📌 Dataset: Shopee Product Matching.
|
|
|
|
|
|
|
|
|
🤗 Hugging Face Datasets: Load and analyze datasets from Hugging Face Datasets. Perfect if you already have a dataset hosted on Hugging Face hub.
|
|
|
|
|
|
|
|
|
🧠 TIMM Embeddings: Compute dataset embeddings using TIMM (PyTorch Image Models) and run fastdup over the them to surface dataset issues. Runs on CPU and GPU.
|
|
|
|
|
|
|
|
|
🦖 ONNX Embeddings: Bring your own ONNX model. In this example we extract feature vectors of your images using DINOv2 model. Runs on CPU.
|
|
|
|
|
|
|
|
See more examples.
Get help from the fastdup team or community members via the following channels:
Community-contributed blog posts on fastdup:
What our users say:
Visual Layer Cloud
Visual Layer offers commercial services for managing, cleaning, and curating visual data at scale.
Sign-up for free.
https://github.com/visual-layer/fastdup/assets/6821286/57f13d77-0ac4-4c74-8031-07fae87c5b00
Not convinced? Interact with Visual Layer Cloud public dataset with no sign-up required.
Disclaimer
Usage Tracking
We have added an experimental crash report collection using Sentry.
We DO NOT collect user-specific information such as folder names, user names, image names, image content, etc.
We do collect data related to fastdup’s internal operations and performance statistics such as total number of images, average runtime per image, total free memory, total free disk space, number of cores, etc.
This help us identify and resolve stability issues, thereby improving the overall reliability of fastdup.
The code for the data collection is found here. On MAC we use Google crashpad to report crashes.
Users have the option to opt out of the experimental crash reporting system through one of the following methods:
- Define an environment variable called
SENTRY_OPT_OUT
- or
run() with turi_param='run_sentry=0'
License
fastdup is licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License.
For any more information or inquiries regarding the license, please contact us at [email protected] or see the LICENSE file.
评论