I submitted a question to the major tech news / discussion site Slashdot a few days ago. It was selected for publication on their site. I got 434 answers so far to my question — I have a lot of reading to do. Here is a link to the question posted on their site.
The question was:
“I’ve managed to consolidate most of my old data from the last decade onto drives attached to my main Windows 7 PC. Lots of files of all types from digital photos & scans to HD video files (also web site backup’s mixed in which are the cause of such a high number of files). In more recent times I’ve organized files in a reasonable folder system and have an active / automated backup system. The problem is that I know that I have many old files that have been duplicated multiple times across my drives (many from doing quick backups of important data to an external drive that later got consolidate onto a single larger drive), chewing up space. I tried running a free de-dup program, but it ran for a week straight and was still ‘processing’ when I finally gave up on it. I have a fast system, i7 2.8Ghz with 16GB of ram, but currently have 4.9TB of data with a total of 4.2 million files. Manual sorting is out of the question due to the number of files and my old sloppy filing (folder) system. I do need to keep the data, nuking it is not a viable option.”
If you stopped by my site to ask me the name of the program I tired, it is called Fast Duplicate File Finder and here is their site.
Any other questions, feel free to use the contact form.