Folder comparison tool?

[Box Brownie]

Hi all

I was unsure how to describe what I seek but need to do the following?

I have backup copies of multiple folders but have realised for reasons unknown some files did not copy across

Therefore I need to 'compare' the files in one location to another to identify the files in source 'A' to the files in target 'B' to identify the missing files in 'B' NB the folder structure in this instance is different so the tool needs to look inside all selected folders in the source and compare to all folders (selected) on the target drive.

I hope that makes sense

I would appreciate any pointers as to the best/good to make that check.

TIA

Edit~
I have found WinMerge that is targeted at text files (as far as I can tell?
Also, FreeFileSync that it seems will 'see' all files

 

[JonathanRyan]

FreeFileSync

 

[JonathanRyan]

Or if your prefer the command line (and who doesn't?) robocopy

 

[ST1]

FreeFileSync

What Jonathan said^
I use it to manage my image backup across multiple drives

 

[hunnymonster]

Robocopy.

 

[Snapsh0t]

Robocopy. It's free as it's part of Windows. I run it in a batch file every night to sync my PC's data folders to my NAS.

 

[Faldrax]

Best comparison tool I've used (and as a software engineer I need to do quite a lot of file/folder comparisons) is Beyond Compare.
Handles a wide range of file types (including images) and (for text based files) allows you to set up rules to define a difference as important or not.
Can also compare remote locations such as FTP sites - so if you upload a website you built via FTP, you can then compare your copy to the remote site, for example.
https://www.scootersoftware.com/

 

[sirch]

WinMerge will compare folders and files and tell you the differences

 

[Box Brownie]

FreeFileSync

Thank you.

Or if your prefer the command line (and who doesn't?) robocopy

My preference is for a GUI

What Jonathan said^
I use it to manage my image backup across multiple drives

My initial need to is compare & identify the missing files, from the source drive, on the target drive.

FWIW they are all raw image files.

Robocopy.

Thanks but as above I am not too enamoured with command line work.

Robocopy. It's free as it's part of Windows. I run it in a batch file every night to sync my PC's data folders to my NAS.

Thank you

Best comparison tool I've used (and as a software engineer I need to do quite a lot of file/folder comparisons) is Beyond Compare.
Handles a wide range of file types (including images) and (for text based files) allows you to set up rules to define a difference as important or not.
Can also compare remote locations such as FTP sites - so if you upload a website you built via FTP, you can then compare your copy to the remote site, for example.
https://www.scootersoftware.com/

I did spot a reference to BC and will have another look.

WinMerge will compare folders and files and tell you the differences

Thank you.
The screenshots looked quite clunky compared to FreeFileSync


Decisions, decisions

PS the reason for the need revealed itself thus.
I copied all, in dated folders, the image files from my travel laptop to a SanDisk SSD.
Once transferred to my desktop I separated out the files based on species (they are my safari pictures). But once completed I realised there were some missing.
Hence to need to compare, a fresh copy, the source files (copied to my Samsung T7 SSD) to differently structured target drive folder of folders.

 

[Box Brownie]

By way of an interim update before installing anything....

I made a folder properties check & comparison:-

Both the SanDisk and the T7 have the same number of files and folders and a quick file check shows that some files are only on those drives in their original folder locations.
However, checking the allocated by species folders there are (edit) 280 files missing.
Now, in checking the files into their species folders I did delete a few obviously bad ones but only a few 10's.

Once I had 'moved the files from the copied dated folders in to the species folders' those dated folders were empty!
So the question remains/appears to be the step of copying from the SanDisk SSD to the desktop did not copy all the files hence the missing ones

Hence, to the upcoming need to use such as FreeFileSync or Beyond Compare (?) to show me the missing files to copy just those safely to a new folder for species allocations.

 

[Box Brownie]

OK

I am trying FreeFileSync and it does not seem to play well with the folders???

The source folder shows the individual folders within the higher level folder with the files listed.
However, the target folder shows none of the sub folders and their contents.
The program in effect is telling me that top level target folder has nothing in it.

I can see the various source and target folders in the lower left window and by picking one I can see the contents in the main window areas.

This does not seem to do what I hoped/thought it would do

 

[Box Brownie]

@Faldrax

I have installed BC and unlike (as far as I could find) FreeFileSync, it was able to open all the sub-folders in both source and target higher level folders.

However, it shows all files on both source & target in Purple i.e. all orphan files

So, is it settings question or because the key use is for sync roles and expects to 'see' the akin folder structure.....................my need is not within its functionality
I have looked at the settings but cannot find anything that might help me?

It is looking like I need to start again from scratch..............with empty subfolders and instead of moving, copy the files across! NB moving made more sense as that progressively isolated the files to allocate & move.

 

[Faldrax]

@Faldrax

I have installed BC and unlike (as far as I could find) FreeFileSync, it was able to open all the sub-folders in both source and target higher level folders.

However, it shows all files on both source & target in Purple i.e. all orphan files

So, is it settings question or because the key use is for sync roles and expects to 'see' the akin folder structure.....................my need is not within its functionality
I have looked at the settings but cannot find anything that might help me?

It is looking like I need to start again from scratch..............with empty subfolders and instead of moving, copy the files across! NB moving made more sense as that progressively isolated the files to allocate & move.

When you first open the comparison it will just show what it finds in the root of each - it has not yet checked what might be in any folders.
Just select the top row, then shift-click the bottom row - everything should then appear selected.
Right click anywhere in the selected section, and choose '=? Compare Contents' (second item in the pop up menu for me).
You'll then get a dialog to select how you want to compare - Binary is good for images, rule based is best for text - click Start and off it will go.
Once done, you should have a nice screen with each line colour coded to indicate if a match or not - you can double click a folder to expand both sides, and double click a file that doesn't match to see the differences (this includes images).
Where you have files present on one side but not the other, you can right click a file or selection of files (using shift-click and alt-click to multi select) and the pop up menu will include copy to left/right.

 

[Box Brownie]

When you first open the comparison it will just show what it finds in the root of each - it has not yet checked what might be in any folders.
Just select the top row, then shift-click the bottom row - everything should then appear selected.
Right click anywhere in the selected section, and choose '=? Compare Contents' (second item in the pop up menu for me).
You'll then get a dialog to select how you want to compare - Binary is good for images, rule based is best for text - click Start and off it will go.
Once done, you should have a nice screen with each line colour coded to indicate if a match or not - you can double click a folder to expand both sides, and double click a file that doesn't match to see the differences (this includes images).
Where you have files present on one side but not the other, you can right click a file or selection of files (using shift-click and alt-click to multi select) and the pop up menu will include copy to left/right.

Hi Jonathan

Thanks for the detailed info but I still only see Purple files in both right & left panes.
It is still not clear to me if, as the folder structures are different between source & target, that that is the issue that I see.

I will try it with two simple folders and see what it presents?

Update ~ I have just tried with two folders one with 3 files and one with 4 files on two different drives (the above attempts were two top level folders of sub-folders on the same drive ~ but as I understand it based on their video tutorial should not make a difference?)
BC opened the folders and showed the 3 files as common and the 4th file in purple as an orphan.

Not sure where that leaves me for now?

 

[Scooter]

You don't really need to "compare". Just copy across any missing files using standard Windows tools:-

robocopy E:\mysourcefolder Y:\mydestinationfolder /E /Z /LOG:BackupLog.txt

Copies only files present on the source and missing from destination /E = copy empty folders /Z use restartable copy mode (recovers from errors and carries on). /LOG: - write output to a new file. This is how I update my backup disks.

 

[Box Brownie]

You don't really need to "compare". Just copy across any missing files using standard Windows tools:-

robocopy E:\mysourcefolder Y:\mydestinationfolder /E /Z /LOG:BackupLog.txt

Copies only files present on the source and missing from destination /E = copy empty folders /Z use restartable copy mode (recovers from errors and carries on). /LOG: - write output to a new file. This is how I update my backup disks.

Hmm!

Does that command line take account of sub-folders?

In the instance I am describing the source folder has 4 subs and within each of those a sub-folder.

The target folder has IIRC 38 sub-folders.

TIA for your further insights

 

[Faldrax]

Hi Jonathan

Thanks for the detailed info but I still only see Purple files in both right & left panes.
It is still not clear to me if, as the folder structures are different between source & target, that that is the issue that I see.

I will try it with two simple folders and see what it presents?

Update ~ I have just tried with two folders one with 3 files and one with 4 files on two different drives (the above attempts were two top level folders of sub-folders on the same drive ~ but as I understand it based on their video tutorial should not make a difference?)
BC opened the folders and showed the 3 files as common and the 4th file in purple as an orphan.

Not sure where that leaves me for now?

Yes, BC will show things as purple if the folder structure is different - but you should be able to 'drill down' into folders to compare the contents, and as you have found you can select any 2 folders to compare (and it will recurse into sub folders of both).
You say the folder structure is different between the source and backup, but I assume at some point there is commonality - simply pick the 'common' folders and compare them.
Note: BC also provides Explorer integration, so you can pick folders/files to compare using the right click menu in explorer.

 

[Scooter]

Hmm!

Does that command line take account of sub-folders?

In the instance I am describing the source folder has 4 subs and within each of those a sub-folder.

The target folder has IIRC 38 sub-folders.

TIA for your further insights

Yes absolutely - it replicates the whole folder structure. 12TB of images in my case. It will copy across any files and folders that exist on the source and don't already exist at the destination - to make the destination look exactly like the source.

 

[gman]

I've been using Syncback Free for a long time and it's a great bit of software and an easy GUI.

 

[Box Brownie]

Yes, BC will show things as purple if the folder structure is different - but you should be able to 'drill down' into folders to compare the contents, and as you have found you can select any 2 folders to compare (and it will recurse into sub folders of both).
You say the folder structure is different between the source and backup, but I assume at some point there is commonality - simply pick the 'common' folders and compare them.
Note: BC also provides Explorer integration, so you can pick folders/files to compare using the right click menu in explorer.

I did put the question to BC support and had this in reply...

"Beyond Compare aligns files and folders based on their names so automatic alignment of items with different names won't occur. There is a Pro edition feature that allows you to specify alignment overrides, i.e. aligning a folder name "Apples" to one named "Oranges" and similar with files. With that you could setup individual overrides to have the appropriate source and target folders aligned.

It's also possible to load each pair of associated folders in a Folder Compare session, and then copy the files as needed. This would involve a bit more work as you would need to do this with each set of folders, but you wouldn't need to setup the alignment overrides mentioned previously and wouldn't require a Pro edition license"

So, it seems that even with the Pro license version my situation would require a (large?) amount of user setup & intervention.

Wil have to consider my options as appropriate?

 

[Box Brownie]

Yes absolutely - it replicates the whole folder structure. 12TB of images in my case. It will copy across any files and folders that exist on the source and don't already exist at the destination - to make the destination look exactly like the source.

Hmmmmm! but I don't want to replicate my source folders i.e. the target folders should not be impacted......as the right now (planned and wanted) my target looks nothing like the source.

 

[Faldrax]

I did put the question to BC support and had this in reply...

"Beyond Compare aligns files and folders based on their names so automatic alignment of items with different names won't occur. There is a Pro edition feature that allows you to specify alignment overrides, i.e. aligning a folder name "Apples" to one named "Oranges" and similar with files. With that you could setup individual overrides to have the appropriate source and target folders aligned.

It's also possible to load each pair of associated folders in a Folder Compare session, and then copy the files as needed. This would involve a bit more work as you would need to do this with each set of folders, but you wouldn't need to setup the alignment overrides mentioned previously and wouldn't require a Pro edition license"

So, it seems that even with the Pro license version my situation would require a (large?) amount of user setup & intervention.

Wil have to consider my options as appropriate?

I use alignment overrides when comparing text (source code) files a lot - it's very useful, and as quick as right click on one side select 'Align With' then right click on the other side where I want it to match to.
I've not used the folder alignment options, but just had a quick look, and it's part of the comparison rules settings - which you can save as far as I know, so you might have to spend a bit of time setting it all up the first time, but should be able to just use the same rule set next time.
Does the trial give you Pro or Standard (I've had Pro for years, so can't recall)?

Out of curiosity, why do you have a significantly different folder structure / naming on your backups?

 

[Box Brownie]

I use alignment overrides when comparing text (source code) files a lot - it's very useful, and as quick as right click on one side select 'Align With' then right click on the other side where I want it to match to.
I've not used the folder alignment options, but just had a quick look, and it's part of the comparison rules settings - which you can save as far as I know, so you might have to spend a bit of time setting it all up the first time, but should be able to just use the same rule set next time.
Does the trial give you Pro or Standard (I've had Pro for years, so can't recall)?

Out of curiosity, why do you have a significantly different folder structure / naming on your backups?

Sorry I thought I explained but only said what there was (in post #16) not why I had the situation.

My source folders are dated ones of the pictures taken on the date/day on safari
The target folders are all where I had separated out the individual species seen regardless of the date taken.
It was only when I reconciled the files counts between the two sets of folders I realised that they did not reconcile.

In other words the target is not in the truest sense a "backup" of the source.

The trial of BC (which I installed as portable) does apparently include the Pro Features.

On the face of it this will be in practice the only time I need to do this process, not withstanding that BC in Standard not Pro might be of use in future

I am taking the time to think about another way to approach my files handling that is more to do with using iMatch and keywording & filtering as required to 'isolate' individual sets of images by species....

 

[Faldrax]

Sorry I thought I explained but only said what there was (in post #16) not why I had the situation.

My source folders are dated ones of the pictures taken on the date/day on safari
The target folders are all where I had separated out the individual species seen regardless of the date taken.
It was only when I reconciled the files counts between the two sets of folders I realised that they did not reconcile.

In other words the target is not in the truest sense a "backup" of the source.

The trial of BC (which I installed as portable) does apparently include the Pro Features.

On the face of it this will be in practice the only time I need to do this process, not withstanding that BC in Standard not Pro might be of use in future

I am taking the time to think about another way to approach my files handling that is more to do with using iMatch and keywording & filtering as required to 'isolate' individual sets of images by species....

OK, that makes sense - and explains why matching/reconciling is such an issue.
I've just had a quick play and I think Beyond Compare can do what you want
If you compare the root of both folder structures, then in the icon bar below the menus, there is a folder icon - select 'Ignore Folder Structure', and it will flatten the view - just matching filenames, with a column for Path (When selected, the icon shows 'Flatten' as it's label, see below)

 

[Box Brownie]

OK, that makes sense - and explains why matching/reconciling is such an issue.
I've just had a quick play and I think Beyond Compare can do what you want
If you compare the root of both folder structures, then in the icon bar below the menus, there is a folder icon - select 'Ignore Folder Structure', and it will flatten the view - just matching filenames, with a column for Path (When selected, the icon shows 'Flatten' as it's label, see below)

View attachment 434694

Many thanks for your researching the subject and I will have a look at it later and see how it behaves

 

[Brentor]

I'm not sure what you use for backup, but would it be easier to start again ie a back up of everything rather than an incremental backup?

NB Looks like your Next Exhibition footnote needs updating.

 

[Box Brownie]

I'm not sure what you use for backup, but would it be easier to start again ie a back up of everything rather than an incremental backup?

NB Looks like your Next Exhibition footnote needs updating.

As mentioned this is not about the backup (I have two backups of all my image files one is a separate internal drive the other is an external portable SSD)

Also, thanks for the heads up re my sig section

 

[hunnymonster]

Hmm!

Does that command line take account of sub-folders?

/E is subdirectories even if empty

 

[Box Brownie]

OK, that makes sense - and explains why matching/reconciling is such an issue.
I've just had a quick play and I think Beyond Compare can do what you want
If you compare the root of both folder structures, then in the icon bar below the menus, there is a folder icon - select 'Ignore Folder Structure', and it will flatten the view - just matching filenames, with a column for Path (When selected, the icon shows 'Flatten' as it's label, see below)

View attachment 434694

Ok

I have done as you suggested and can see on the target side some in 'red' i.e. changed (newer or different) and on the left source side various in purple

Now, the question is..............as previously mentioned...................is it possible to filter the files based on them being purple and it so copy them en bloc to a designated folder?

 

[Faldrax]

Ok

I have done as you suggested and can see on the target side some in 'red' i.e. changed (newer or different) and on the left source side various in purple

Now, the question is..............as previously mentioned...................is it possible to filter the files based on them being purple and it so copy them en bloc to a designated folder?

Yes (I think)
On the Icon bar, click the drop down arrow on the 'Diffs' button (to the right of the Flatten button, between 'All' and 'Same') - select 'Orphans', 'Left Orphans' or 'Right Orphans' - this will filter the display accordingly.
You can then select the files you are interested in, right click then from the menu choose 'Explorer', then 'Copy'.
Then open Windows explorer, browse to where you want the files to go, and Paste them in

(There may be other ways to do this, that's just the one I found first!)

 

[Box Brownie]

Yes (I think)
On the Icon bar, click the drop down arrow on the 'Diffs' button (to the right of the Flatten button, between 'All' and 'Same') - select 'Orphans', 'Left Orphans' or 'Right Orphans' - this will filter the display accordingly.
You can then select the files you are interested in, right click then from the menu choose 'Explorer', then 'Copy'.
Then open Windows explorer, browse to where you want the files to go, and Paste them in

(There may be other ways to do this, that's just the one I found first!)

Jonathan

Thank you once again for your efforts & support, it is much appreciated

I found that your suggestion above in choosing RHS (Source) showed 100% of all files as Purple Orphans

So having re-reviewed the un filtered list I could identify blocks of files to copy into a folder but some were all jumbled up with what looked like duplicates. I rechecked the source folder selection and realised it included a folder of mixed files ~ doh! I cut that and put it elsewhere and refreshed BC and that made the RHS more clear.....I have now worked though the listing and copied the blocks of files for further examination as needed and putting in the species files.

Thanks to you I am that bit nearer to resolving, fingers crossed, the problem

Hopefully sometime in the next couple of weeks I will process a few and post in OM system sub fora. I will ref you in to see them

 

[Scooter]

Hmmmmm! but I don't want to replicate my source folders i.e. the target folders should not be impacted......as the right now (planned and wanted) my target looks nothing like the source.

I have backup copies of multiple folders but have realised for reasons unknown some files did not copy across

Therefore I need to 'compare' the files in one location to another to identify the files in source 'A' to the files in target 'B' to identify the missing files in 'B' NB the folder structure in this instance is different so the tool needs to look inside all selected folders in the source and compare to all folders (selected) on the target drive.

If your aim is to make a backup - I can't think of a better way.