Files corrupted when copying to external usb3 hdd

[E<>MC^2]

Do you use Microsoft Security Essentials antivirus by any chance?

I had exactly the same problem more than a year ago and it really annoyed me as it corrupted a 'seemingly random' selection of my valuable backup files, and without verifying afterwards, it was impossible to tell if the file was corrupted or not as Windows did not complain about any errors. There were no logs or errors about disk controller faults or anything like that. It was only because the files were compressed, and the fact that they failed the checksum checks that I knew the files were corrupted.

In my case, it occurred with both USB2 (4GB) pen drives, and also portable USB2 portable hard drives (between 320 MB and 1TB) on Windows 7 (64 bit), and all using the 'safe removal' option. I was 95% sure it was not the drives or the cables as they would work fine on other machines.

The problem only seemed to be occur more when copying large file (at least 256MB, but usually larger when using hard drives). Maybe MSE locks the file while scanning and this interferes with the copying process, just a wild stab in the dark 'no-evidence-kind-of-guess'.

After some experimenting, I discovered that removing MSE solved the file corruption problem copying using Windows Explorer. That was enough for me. After this point, I stopped using MSE, and since then no more problems ever with file corruption! Just to round it off, no problems before using MSE, no problems after removing it, but problems while it was installed.

I don't think it's solely MSE's fault, as I can't believe the millions of other users would not have encountered the problem, but maybe the combination of that with my disk controller perhaps is responsible. Not suggesting your issue will be the same, but thought it was worth sharing.

BTW, while experimenting I did try TeraCopy and the copies were verified to be good. I just preferred to use the built in explorer.

Yes I am using MSE. The thread I listed above also mentions MSE as a contributing factor in combination with the other options I mentioned above.

My first approach would be the other methods, and not removing MSE.

Thank you for detailing how removing MSE helped.

[hello_hello]

BTW, while experimenting I did try TeraCopy and the copies were verified to be good. I just preferred to use the built in explorer.

I just had a quick play with TeraCopy for the first time, and it seems to integrate with Explorer pretty well. It appears you can set TeraCopy to handle all the normal copy and paste and drag and drop operations, and I assume it does it's thing in the background so you're just copying and pasting as you normally would.
Or instead of dragging and dropping with the left mouse button you can do it with the right, and TeraCopy is included in the usual list of copy options.

I use xplorer² myself, as Windows Explorers sucks, and TeraCopy integrates with it pretty well, as xplorer² also uses the Explorer context menu. Not that I have an file copying problems I'm aware of (I'm using XP), but I'll play with TeraCopy some more.

[p_l]

Actually, I'm playing around with TeraCopy for the first time myself, too, and I like it. Yes, it integrates well into shell / right-click options, but it also works with left mouse dragging and dropping, and you initially get this:

[Lakesidex]

Me and a buddy have documented this same problem on Win7 x64 using Biostar MB's with the Asmedia USB 3.0 chipset.

Two solutions that worked for me were:

1) purchased a Vantec 4-Port SuperSpeed USB 3.0 PCIe Host Card with Internal 20-Pin Connector UGT-PC345. Worked perfectly on the same PC that was corrupting data through its built-in Asmedia USB 3.0 ports. This PCI card uses the NEC/Renesas uDP720201 chipset. Probably any card using that chipset would have worked.

2) Changed the Policy for the USB drive in device manager to "Better Performance" instead of "Quick Removal". This fix worked on my PC as well as my buddy's for the built-in Asmedia USB 3.0 ports. This one is a simple fix that anyone should be able to use.

Good luck all with this ugly problem.

[fritzi93]

Me and a buddy have documented this same problem on Win7 x64 using Biostar MB's with the Asmedia USB 3.0 chipset.

Two solutions that worked for me were:

1) purchased a Vantec 4-Port SuperSpeed USB 3.0 PCIe Host Card with Internal 20-Pin Connector UGT-PC345. Worked perfectly on the same PC that was corrupting data through its built-in Asmedia USB 3.0 ports. This PCI card uses the NEC/Renesas uDP720201 chipset. Probably any card using that chipset would have worked.

2) Changed the Policy for the USB drive in device manager to "Better Performance" instead of "Quick Removal". This fix worked on my PC as well as my buddy's for the built-in Asmedia USB 3.0 ports. This one is a simple fix that anyone should be able to use.

Good luck all with this ugly problem.

Welcome to the forum.

Glad it worked for you. BTW, I have Asmedia USB 3.0 ports on this computer as well, policy set to quick removal (default).

It was always my understanding that quick removal is safest, because it disables write caching. Once the transfer, deletion, whatever, is completed, it's safe to remove the drive. Slower, but safer.

However, I wonder if some people seeing corruption have gotten in the habit of treating USB external hard drives as "hot swap", and fail to use Safe Hardware Removal. Regardless of the policy setting. There is by no means unanimity on this, even though many "experts" say it's okay with quick removal policy.

Here's just one link on this subject, and I've excerpted one of the comments:

http://www.cnet.com/how-to/quickly-remove-usb-devices-without-using-safe-removal/

"According to the Quick Removal (default) policy statement, since this disables write caching, I should be able to disconnect my USB disk drive from the USB port without having to go through the USB disk drive ejection process as part of the Safely Remove Hardware ...

HOWEVER, I have found out the hard way that USB portable drives (e.g. Toshiba Canvio 1.5TB and many others) will not use the "Quick Removal" policy no matter whether it was set by default (essentially always is) or manually set. Just does not work. You have to use the "safe to Remove" op sys feature. If you don't, eventually the drive becomes inaccessible totally. Formatting the drive is the only cure (and you lose whatever you had on it forever.) Many respected gurus on websites like this one have been claiming this makes the devices "hot-swap" and that is NOT TRUE, at least not for many of these drives over 1TB in size. No idea why the size matters ...

Take a look at the reviews on the Amazon.com site for any/all the portable USB drives over 1TB. Hundreds of complaints ! Same on the Toshiba site."

[Lakesidex]

delete

[Lakesidex]

[QUOTE=fritzi93;2380723]

It was always my understanding that quick removal is safest, because it disables write caching. Once the transfer, deletion, whatever, is completed, it's safe to remove the drive. Slower, but safer.

However, I wonder if some people seeing corruption have gotten in the habit of treating USB external hard drives as "hot swap", and fail to use Safe Hardware Removal. Regardless of the policy setting. There is by no means unanimity on this, even though many "experts" say it's okay with quick removal policy.

Here's just one link on this subject, and I've excerpted one of the comments:

http://www.cnet.com/how-to/quickly-remove-usb-devices-without-using-safe-removal/

"According to the Quick Removal (default) policy statement, since this disables write caching, I should be able to disconnect my USB disk drive from the USB port without having to go through the USB disk drive ejection process as part of the Safely Remove Hardware ...

HOWEVER, I have found out the hard way that USB portable drives (e.g. Toshiba Canvio 1.5TB and many others) will not use the "Quick Removal" policy no matter whether it was set by default (essentially always is) or manually set. Just does not work. You have to use the "safe to Remove" op sys feature. If you don't, eventually the drive becomes inaccessible totally. Formatting the drive is the only cure (and you lose whatever you had on it forever.) Many respected gurus on websites like this one have been claiming this makes the devices "hot-swap" and that is NOT TRUE, at least not for many of these drives over 1TB in size. No idea why the size matters ...

Take a look at the reviews on the Amazon.com site for any/all the portable USB drives over 1TB. Hundreds of complaints ! Same on the Toshiba site."

I agree that your analyses makes sense and is how it is suppose to work. I always use "safe to remove" op sys feature for all USB devices. But we are talking about copying an 8GB MKV file to a USB device and not getting a MD5 match. I did ten different tests copying large files to the 1TB Toshiba Canvio with the policy set to "quick removal" and every time the MD5 sigs didn't match. When I changed the policy to "better performance" the corruption went away. I was able to copy 10 more files and get perfect matches. I was also able to get perfect Md5 matches using the Vantec PCI card.

The way I found this error was by playing backed up videos of of the USB drive and getting corruption in the video. Doing Md5 scans confirmed the corruption. That is when I began researching and testing. In my opinion this is a Win7 x64 bug. You shouldn't be able to flip a switch like this and break the abilty to do basic uncorrupted Windows copies. At the least windows should throw you an error saying the copy is corrupt.

More info here..
https://social.technet.microsoft.com/Forums/en-US/13a7426e-1a5d-41b0-9e16-19437697f62b...um=w7itproperf

[fritzi93]

You're right, of course. What I speculated on (and it's no more than speculation) didn't pertain to the original issue. Still, is it not possible this file transfer mismatch issue can arise on an external hard drive already corrupted? (Not in your scenario specifically).

It does sound like a Win7 bug, doesn't it?

[Lakesidex]

You're right, of course. What I speculated on (and it's no more than speculation) didn't pertain to the original issue. Still, is it not possible this file transfer mismatch issue can arise on an external hard drive already corrupted? (Not in your scenario specifically).

It does sound like a Win7 bug, doesn't it?

My fault, I left out some information.

Yes you're right it could arise from a hard drive already corrupted.

I forgot to mention that I did a full CHKDSK on the drive it it reported zero errors.

Thanks for pointing that out... just trying to help others

[kyrcy]

The only workaround I was able to find so far is to use TeraCopy. I leave unchecked the "Use system write cache" box and check the "Always test after copy" box just to be sure.

[Lakesidex]

The only workaround I was able to find so far is to use TeraCopy. I leave unchecked the "Use system write cache" box and check the "Always test after copy" box just to be sure.

Did you try changing the Policy to"Better Performance"? Just curious if this works for you. What USB chipset are you using?

I've heard that TeraCopy is a good option but I haven't tried the program. I do plan on trying it though, just to see how it works. But the Policy change worked for me and my 1TB Toshiba Canvio. I've also had no problems with the Vantec USB 3.0 PCI 1x card.

[kyrcy]

At some point I tried changing the Policy to"Better Performance" but from what I remember it did not solve the problem completely. The problem also was happening when using a USB 3.0 PCIE card with Renesas chipset.

[bevills1]

I've read about problems using Windows backup/restore and continue to use Seagate Discwizard to backup/restore. I've also had Windows defrag lockup and not complete, and I now use Auslogics defrag instead. My point is some Windows apps seems to have issues, and there are third party apps that resolve those issues. Personally I'd use Teracopy to resolve the copied files corruption problems if that's what works.

[Lakesidex]

I've read about problems using Windows backup/restore and continue to use Seagate Discwizard to backup/restore. I've also had Windows defrag lockup and not complete, and I now use Auslogics defrag instead. My point is some Windows apps seems to have issues, and there are third party apps that resolve those issues. Personally I'd use Teracopy to resolve the copied files corruption problems if that's what works.

So at this point I would suggest a troubleshooting strategy:

1) Change the Policy for the USB device in question from "Quick Removal" to "Better Performance " and test to see if that solves the problem.

If this does not work

1) Try TeraCopy and see if that is a workable solution for the user.

2) Roll the dice and see if a USB 3.0 PCI card with a different chipset than the one giving problems corrects the solution.

[Seeker47]

You're right, of course. What I speculated on (and it's no more than speculation) didn't pertain to the original issue. Still, is it not possible this file transfer mismatch issue can arise on an external hard drive already corrupted? (Not in your scenario specifically).

It does sound like a Win7 bug, doesn't it?

I'm convinced that it is. Incidentally, I have seen this "corrupted copies" issue over e-SATA as well, and even a few times to a 2nd. internal HDD ! But only on x64 W7 -- can't recall a single incident under 32-bit W7.

TeraCopy has been a most welcome workaround, and there's a version that is free for personal use. Of necessity, I've gotten into the habit of doing all my copying in x64 through it. This is slightly less convenient than using my file manager program (which does have its own file compare feature), but the peace of mind is worth it.

More evidence of this being a W7 bug is that I don't run into any problems making Acronis TI partition backups, which pass the 'Verify' run after creation. These will have to be perfect, in order to be of any use. Acronis is an outside-the-OS app, which boots its own mini-OS from the CD (Linux-based, I would assume). The image can be recorded to either an internal or an external drive. W7 has no involvement in this process, and it works.

[Seeker47]

I've read about problems using Windows backup/restore and continue to use Seagate Discwizard to backup/restore. I've also had Windows defrag lockup and not complete, and I now use Auslogics defrag instead. My point is some Windows apps seems to have issues, and there are third party apps that resolve those issues. Personally I'd use Teracopy to resolve the copied files corruption problems if that's what works.

So at this point I would suggest a troubleshooting strategy:

1) Change the Policy for the USB device in question from "Quick Removal" to "Better Performance " and test to see if that solves the problem.

If this does not work

1) Try TeraCopy and see if that is a workable solution for the user.

2) Roll the dice and see if a USB 3.0 PCI card with a different chipset than the one giving problems corrects the solution.

I have some older Shuttle XPC models, which did not then offer USB-3, so I had to try some expansion card retrofit options. (I like these computers too much to replace them with something later, and they still cover my needs well.) What I've found is that there are a bunch of vendor & card models, but maybe just 3 or 4 chipsets that keep turning up in them. There is a common NEC one, etc. Some of these proved to be flat out incompatible with the computer's BIOS, so it was game over. Most of the cards are PCI-e, which is also bad news for my SP35 Shuttle, which could only take a plain PCI type. Those cards are hard to find, and if you do find them they can be $65. or more, which is ridiculous for an experiment that might well not work out anyway. (And even if it did work, some sources are saying that the speed increase over USB-2 to be had there would be rather disappointing -- for other reasons, like what speed the bus can accommodate.) I had a bit more success with a PCI-E one that I put into a SX48 Shuttle. The speed increase there did seem to be worth it. However, one of the two ports on it seem to have burned out permanently, during some strenuous Copy job. I was just trying to supplement the e-SATA option that fortunately is already built-in.

But I still need to use TeraCopy under 64-bit W7 !

[kyrcy]

I have recently noticed that archives of large size that are created on external hard disks (and not just copied to) are often corrupted and that verification fails when testing them. Changing policy to"Better Performance" (without enabling write-caching) seems to help, but further testing is needed to verify if it solves the problem completely (under this scenario) or not.

[Lakesidex]

I have recently noticed that archives of large size that are created on external hard disks (and not just copied to) are often corrupted and that verification fails when testing them. Changing policy to"Better Performance" (without enabling write-caching) seems to help, but further testing is needed to verify if it solves the problem completely (under this scenario) or not.

Definitely keep us informed with how your testing goes.

[boboyboboy]

My experience with file transfer using Windows COPY utility from ATA host HDD to external USB3 HDD (any enclosure, any size HDD from 3TB to 5TB) has been frequent disconnects when the file transfer was one or multiple large files (> 2GB). My OS is Windows 7 x64.

My solution, which has not failed, is to use TotalCommander. First, it provides detailed information about the files or folders (directories) to be moved. After the transfer is complete, the number of files and folders is in the result information. If there are residual files (my have been added to a folder after the transfer started), the information about the files/folders not moved is displayed. There has never been an instance where the copy was incomplete.

You can download and use TotalCommander without charge, but I find the utility so versatile and reliable that I have sent a contribution to the developer.

Here is the link:

http://www.ghisler.com/

With respect to Teracopy: I have it and I do not use it.

[kyrcy]

Personally I don't like TotalCommander. I much prefer Far Manager which is free. But I need to test it further with copying files.

[Seeker47]

The CRC comparison test that TeraCopy gives you after each Copy or Move operation is an essential feature. (Any of the other hash algorithms -- as seen in the free HashCalc utility, which gives you 10 to choose from -- would be fine too, but you need something, in order for these file copies not to be a waste of time.)

[Dishe]

I'm so glad I found this thread. My job requires me to work on a lot of media editing, and I often edit video I recorded stored off a USB drive. Lately, I'm finding blank/missing frames which aren't blank/missing on the original media.

I used to think it was my card reader going and giving intermittent errors, so whenever it happened I would just copy it again. I'd perform some MD5 checksum tests, and found that while most of the files checked out OK, the ones with bad data sure enough did not match, despite being the same size and everything. A re-copy usually solved the problem. But then I had the same issue copying from one external USB drive to the other!
I was going crazy, thinking my USB cables were bad (switched, its not), the drive was failing (chkdsk and other utilities could not find anything wrong), or that my OS was corrupted somehow... boy am I happy to find out its just Windows!

Wait, no, that's actually TERRIBLE.

So, I've got a few burning questions I hope you folks can help me answer:
1) Does the problem still exist in Windows 10? I have installed win10 on my other personal computers, but I've pushed off upgrading my workstation because I have a lot of legacy tools that are no longer supported for my job, some of which took a lot of coaxing to make work in my current Win7/x64 setup. I'm afraid of "fixing what ain't broke" and having a headache with those tools. But if the Windows copy command is broken, that's a compelling reason to move on!

2) It seems like Teracopy is a workaround, but I also use FreeFileSync to incrementally back up any changes to my data nightly. FFS uses the Windows copy routine under the hood, so I'm guessing I've been corrupting my backup all this time (BOOOO!). Is there a backup utility, perhaps one that supports incremental backups, that can use Teracopy's protocol? Or can Teracopy replace Windows' somehow, so that all utilities that access it end up using Teracopies instead? Long shot, I know, but couldn't hurt to ask.

3) Assuming that I'm stuck with FFS for my incremental backups (and probably other similar backup tools that would rely on Windows' copy routine), what can I do to make windows' copy more reliable?
-kyrcy sounded hopeful changing USB settings to "Better Performance", but he didn't get back to us with the results of his further testing. I would gladly do that if it works, as the drive is permanently on my desk and plugged in. It does not get unplugged and shuffled around like a USB thumb drive.
-Would uninstalling Security Essentials fix it? I rather like MSSE because of its small footprint that doesn't bog down my machine- I don't even know what I would replace it with, but is that a confirmed fix?
-new hardware? I actually was thinking about getting a USB3 PCI card to speed things up (I have a lot of USB2 ports on this tower), but that seems like a crap shoot.

I'm open to all and any advice! If it were as simple as buying a new computer, I'd go that route, but my work tools took a long time to get set up properly so I'd much rather do whatever I can to fix this one rather than start over and deal with that whole headache again.

[Dishe]

Has anyone confirmed that exFAT doesn't have this problem? It is mentioned earlier that it seems to effect NTFS drives, but I'm not sure if that's just because the drives tested happened to all be NTFS (since that's the default), or if anyone actually tried a non-NTFS and confirmed it worked.

[Seeker47]

I'll respond to what points I can, re posts #52 / 53:

It is very strange how this problem seems to be confined to 64-bit W7, but I've simply not encountered it in 32-bit W7. (I maintain several computers.) NTFS may be a red herring here. I copy plenty of stuff to FAT-32 partitions as well. The thing is, the higher you go in file size -- say, above roughly 1.3Gb. file sizes, but particularly once you go over 2 Gb. size files -- the more likely you are to be bitten by this bug. I regularly copy DVD file structures, but their components top out at 1Gb. size VOBs. (The file-size limitation for FAT-32 pretty much dictates that you would not be working with jumbo .MKV files there, anyway.)

I presently have one Win 8.1 Pro x64 testbed system, but have not done enough on it relevant to this discussion for me to comment. So far, I've been steering clear of 10 for multiple dealbreaker type reasons.

Terracopy does have a paid, pro version available. I don't know what features it may add, other than a "memory" of your last "X #" of copy operations for editable re-use, but you might want to find out.

I've tried retrofitting older rigs with USB-3 via expansion port cards. The tally to date is 1 limited success (with only moderate speed improvements), and several failures, due to hardware compatibility issues. There seem to be many near-identical expansion cards on the market, under various brands, but unfortunately with too many of them based on the same underlying chipset. A better variety in chipsets might have yielded different results. A similar thing happened in my experience with regard to attempts at e-SATA retrofits. You get the appropriate driver, but when the MB then declines to boot up, it is clearly trying to tell you something !

[aedipuss]

i've had the problem with a wd 4tb external usb3 drive since i got it. i use it with a desktop win7 machine and a win8.1 laptop as it's main storage as it's only got a small ssd. all i can say is never ever do anything that might interrupt the usb connection or ask a usb hard drive to do 2 things at once. if you are copying files to or from it, walk away and don't have the computer do anything else until it's done.

[Paramonte]

Ok, windows 10 64 bits here and 1648 days after the last post
This week got two enclosures (usb) and got two 6 TB Toshiba disks too put in. Setup and test with check disk showed no bad sectors in both drives. No idea what the usb chipset is inside the enclosures, but will find out.

I performed a backup (using copy-paste) to both drives from folders in my PC. Each folder about 1.5 TB . Individual files with sizes ranging from 20 MB to 8GB.

I report this rather strange behavior:

Some of the bigger files (around several GB) got corrupted...an error was issuedr if I tried to open them in Matlab. However a few minutes latter, under the same operation they would open. Then one other of these files would open ok in three successive tests, and would report error from then on.

This is medical data, extremely important, thank God I have two backups: one in the PC other in the google drive. I retrieved the same problematic file from the google drive and worked OK in matlab in several attempts.

Sorry, I know I should have performed a checksum of both but did not, my bad. Will do now with TeraCopy

Regarding the file corruption, seems that a whole sector is bad and depending on how the file reader is initialized it will bump on the defective sector or not. Like rolling the dice.

Two things to consider: I might have not used always the safe usb removal op, and since the enclosure (Port Connect is the brand) have an hard switch (something I like) I might have switched them off a couple of times without the safe removal op.

This has been a TOUGH week on me...

[Seeker47]

This is yet another good example of why I take a very liberal stance on the Older Threads issue: newer and relevant info comes to light on a specific subject, and unless the thread is already well over a hundred pages long, I'm not at all sold on thread proliferation on the same topic being a net plus. Just more stuff to search through, and maybe still not find what you were looking for. (In that regard, having a browser bookmark for threads you have participated in at some point -- on this or any forum -- can become a valuable shortcut.)

I think aedipuss was quite right in post #55, above. Even though I mostly use older computers and OSes (several, in fact), and happen to despise Win-10, I now have to take it into some consideration, seeing as I recently acquired a couple of somewhat less old systems, running i5 & i7 with pre-installs of W-10. (I may still switch those over to the Enterprise LTSB channel, if that frees you up from frequent updates with significant changes, and still offers the ability to opt out of particular updates.) But in terms of the quantity of experience with W-10, so far I'm probably one of the last people you'd want to ask for any knowledgeable advice on what the OS does or does not do. I really hate the way MS feels compelled to keep changing the name and location of common components between OS versions: just another banana peel and time-waster tossed in our path. And I bet the compatibility issue with many apps beloved by VH members is likely a lot worse than tends to be discussed. I see lots of good reasons to stick with 7 indefinitely. If at some point it becomes prudent to disconnect such systems from the internet for security reasons, so be it. I'm in a good position to platoon different systems for specific purposes. Because I've long been all in with small form factor rigs, swapping them in & out is only a minor inconvenience.

The other day, I was running a MakeMKV job with an external Blu-Ray burner and external, portable USB-2. (Only the later rigs I mentioned have USB-3, and those are not connected up at the moment.) This was with my standard W7U x64. Something weird happened, that is the closest I've seen to what you reported (for W-10) in quite some time. The job apparently finished, and nearly all of the files received a CRC match from TeraCopy -- that Verify after Copy being a stellar feature it provides, and I think the actual copying goes faster than by other scenarios as well -- but TeraCopy also reported the job as not completed, as the drive was no longer detected. I could not even dismiss the drive, and thought that I might have to reboot. A USB port having gone kerflooey, perhaps ? I have seen that happen once or twice over the years. I had some concern that the MKV files produced by this operation might not be good, but so far there is no indication of that. Of course, if you run into bad sectors on a HDD, all bets can be off.

This may or may not be of value when it comes to W-10, but I've been a big fan of the Antec Veris MX-1 enclosures. They've been off the market for some time, but are still often seen on eBay, even sometimes as sealed-box, New Old Stock. They have a dual-interface -- USB-2 or e-SATA, your choice. Well made, with a fan inside the enclosure for cooling. I keep several, usually outfitted with a 1 or a 2 TB. WD Black. (Don't know if they'll support anything larger -- maybe not.) If your computer has e-SATA, this should be much closer to the speed of USB-3, and -- hopefully (?) -- does not share in this Windows USB glitch. But I still wouldn't push the multitasking unduly, as aedipuss cautioned.