Big problem with MP4 videos after using USB 3.0 - please help!!

[lg1988]

Hi,

I decided to back up some MP4 videos using a USB 3.0 stick, so I then deleted them on my computer. There was no problem with any of the videos when they were on my computer but when I try to play them now, the picture is all messed up like this http://oi67.tinypic.com/2d6kyue.jpg

I would really appreciate it if someone could help me. I have been trying to fix it but nothing seems to work

Thank you!

[smrpix]

Try copying one or two back to your computer (internal HD?) and see if the problem persists.

I have been trying to fix it but nothing seems to work

What have you tried?

[sneaker]

Can you try on a different PC or at least USB port (many mainboards have 2 different USB chipsets to allow lots of ports)? If you are lucky the problem is a reading problem. If the same problem can be seen on a different PC it means the files were corrupted on write which means you'd be SOL.

[lg1988]

Try copying one or two back to your computer (internal HD?) and see if the problem persists.

What have you tried?

I've tried that and they're exactly the same. I've tried different 'repairing software' and I just end up with a message saying that there's no problem with the file. I've opened the mp4s on VLC Player, Windows Media Player etc. but the problem remains.

Can you try on a different PC or at least USB port (many mainboards have 2 different USB chipsets to allow lots of ports)? If you are lucky the problem is a reading problem. If the same problem can be seen on a different PC it means the files were corrupted on write which means you'd be SOL.

I will try to see if they'll play on a different PC

I won't hold my breath. Thank you both for your help

[abolibibelot]

1) I never, ever, delete any file I just copied onto another storage device without first verifying the integrity of the copy (especially if I don't have a backup copy somewhere else). If it's just downloaded movies it's not a big deal, but for more important files, relying on hardware and software always working as they should can lead to disasters. Some free tools to do that, which I regularly use and can vouch for :
- TotalCommander (“synchronize” function, compare by content -- I've found that it could give false negative, when system memory is low or the computer is busy with other tasks, but I'm pretty confident it won't give false positive, i.e. if it says two files are = then they're =)
- WinMerge (compare by full content -- probably the most practical for such purpose : in “tree” mode it will indicate if two entire trees of files are identical, and contrary to CRC32/MD5 based methods below it analyzes the files on both sides simultaneously therefore it's generally much faster)
- DoubleKiller and the likes (well, DK pro ain't free, but can be found... the free version is way slower ; there are of course other efficient duplicate files detector but this one is my favorite, simple yet intuitive, efficient and with enough options to deal with almost any situation)
- MD5Checker and the likes

2) If you didn't do too many write operations on the storage device where those files were located originally since you erased them, you may still be able to recover them with a good data recovery software. Recuva is free and indicates if the files it detects were overwritten or not (not perfectly reliable though).
Maybe they were already corrupted before they were copied and you didn't know. When did you last play them flawlessly ?

3) Your USB stick may be corrupted, or even counterfeit. There are tools to test them. (H2testw, ChipGenius, USBFlashTools -- I tried those a few years back when after I had bought a counterfeit SD card, but don't remember which does what and how well.)

4) Do these garbled frames appear right from the begining, or at a particular place / time ? What repairing softwares did you try ? If you still can play them, the header must be fine. Opening them in an Hex editor may give some clues to understand what went wrong, but it requires some experience and a lot of patience. If you can recover at least one of those files and it plays fine, you can compare it with the one copied and corrupted, to see where differences appear and possibly identify what they are. But that's a strange case : MP4 are complex files, with picture and sound intertwined, it would be very unlikely that a random data corruption would result in only the picture being affected, and only partially so.

[turk690]

If you are using windoze explorer (windoze 7 or thereabouts), there are random problems copying large files (>100MB) to and from USB drives. There are a plethora of issues concerning explorer itself, the USB chipsets used on the motherboards you are using (and the drivers for them), and USB drives themselves. Basically, the file can get corrupted, as an MD5 checksum will reveal that the copy doesn't have the same hash as the original file. (If corruption did not occur on the file header areas some programs will indicate nothing wrong with the copy.) To get around this I have installed teracopy, which makes bit-perfect copies all the time, across multiple incarnations of different USB drives and USB chipsets. It has a built-in MD5 checker, which can be set to automatically verify if the copy is identical to the original (adds to the copy time; but since I'm always assured of bit-identical copies I mostly turn the verify feature off).

[abolibibelot]

...but since I'm always assured of bit-identical copies I mostly turn the verify feature off).

There's no such thing as “being always assured of bit identical copies”, regardless of the software used. If you don't use the MD5 checker you could still end up with a corrupted copy, mostly because of a corrupted storage device, buy also for other weird reasons.
For instance, I used Synchronize It! 3.5 which I thought was perfectly reliable, beside being one of the very few Windows file/folder copying software which retain all the timestamps (even modification / creation / access dates of folders), but I discovered when doing such verifications that it systematically corrupted files which were downloaded with specific download managers (like JDownloader 2, FlashGet... opened in an hex editor those files appeared to be empty -- filled with zeroes -- after exactly 25000 bytes, yet the “compare” function included in Synchronize It! wouldn't report any problem). I informed the author : it turned out that his program was indeed corrupting files with the “sparse” attribute, and he provided me with a fixed version (which I hope he then released publicly (1)).
Then I also identified a bug which affects the highly regarded Robocopy and FastCopy (and Synchronize It! in a slightly different way ; I didn't test TeraCopy but I'm pretty sure it would be affected too) : I had a folder of pictures obtained with a data recovery software where some files (as I found out later) had the same 8.3 short name (albeit a distinct regular name), and it happened that, for each of these pairs, the copied file was the other one with the same 8.3 name, while the file with that regular name in the source folder was missing in the destination. In this case it worked if I first copied the missing file only with Synchronize It! (unchecking the one which would replace it otherwise), then the other one, but if I re-synchronized the folder with the almighty Robocopy (well, not so much in that particular case) it would be messed up again.


(1) Well, apparently he did not, which is a shame... the version currently available on the download page is from 2009.
http://www.grigsoft.com/wndsync.htm
I can still recommand this software which is very easy to use and usually reliable, and, again, keeps all the timestamps (wheareas most copying / synchronizing applications will at least change the folder timestamps), but beware that it can corrupt certain files (and the “sparse” attribute isn't one that's displayed in the “attributes” column in Windows Explorer* so it's not so easy to check). Or send a message to the author to request the updated version, 3.5.0.1073 (from october 2015).
And anyways, always check the copy afterwards regardless of the software used, especially before deleting the files from the source.

* EDIT : True for Windows XP, but it does appear in Windows 7 as the “P” attribute.

[hello_hello]

I think every thread I've read involving corrupted files when copying to another drive (mainly Win7 and USB3) the solution seems to be to use TeraCopy. The way I understand it, it doesn't use Windows for copying files, so whatever the Windows problem causing corruption, it doesn't effect TerraCopy. It integrates into Explorer so using it isn't all that different to using Windows copy and paste etc, and I think it can be configured to replace the Windows copy/paste functions entirely.

I'm pretty sure TeraCopy can "verify"after it copies, but once you're confident the drive you're copying to doesn't have problems, you probably wouldn't need to keep verifying unless you're especially paranoid. I haven't used TerraCopy for a while myself.... because XP doesn't have copying issues.

For instance, I used Synchronize It! 3.5 which I thought was perfectly reliable, beside being one of the very few Windows file/folder copying software which retain all the timestamps (even modification / creation / access dates of folders), but I discovered when doing such verifications that it systematically corrupted files which were downloaded with specific download managers (like JDownloader 2, FlashGet... opened in an hex editor those files appeared to be empty -- filled with zeroes -- after exactly 25000 bytes, yet the “compare” function included in Synchronize It! wouldn't report any problem). I informed the author : it turned out that his program was indeed corrupting files with the “sparse” attribute, and he provided me with a fixed version (which I hope he then released publicly*).

Your the first one! I've wasted huge chunks of my life surfing the internet but you're the first person I've come across who also uses Synchronize It!
Mind you I'm still using version 2.82. I think I tried version three at one stage but stuck with version two. It was so long ago, I can't actually remember why. Laziness maybe....
I probably bought it for the same reason you did.... although about 15 years ago..... and for all that time it's worked flawlessly. Maybe version 2 doesn't have the same verification abilities as version 3. I'd have to check as I've only ever used it for syncing, and I've not used the download managers you mentioned. The only oddity I've found in 15 years.....

In order to customize a folder in Windows (with a unique icon etc) the folder has to have a read only or system attribute. When Synchronize It! deletes files from customized folders it's read only or system attribute is often removed, probably by Windows itself (I've not been able to work out why), and I have to manually reset the attribute. No big deal really, just a minor annoyance.

I run Synchronize It! every day. A configuration to sync the files between two PCs, another to then sync one of the PCs to an external hard drive. One for syncing my MP3 folder with my MP3 player, and one for backing up as many program configuration files as possible. I'd be a disorganized mess without it.

It might sound odd but I'm actually a little excited to meet another Synchronize It! user. After all this time it's almost the software equivalent of first contact for me. There is someone else out there....

PS How would I go about creating a sparse file to sync for testing? I'm not sure I've seen one. Are they a new Windows thing or an NTFS thing? Can XP can play too?

[abolibibelot]

I'm pretty sure TeraCopy can "verify"after it copies, but once you're confident the drive you're copying to doesn't have problems, you probably wouldn't need to keep verifying unless you're especially paranoid. I haven't used TerraCopy for a while myself.... because XP doesn't have copying issues.

What are the known problems of other (Windows I suppose) systems related to file copying ? I resisted making the jump for a veeery long time but now I'm forced to (if only to access my 3+TB hard drives). So currently I'm using Win7 which has quite a few things I find very irritating. If file copying is known to be unreliable with it that would make another one (even though I rarely used direct copy function in WinXP ever since I became “especially paranoid”).

Your the first one! I've wasted huge chunks of my life surfing the internet but you're the first person I've come across who also uses Synchronize It! Mind you I'm still using version 2.82. I think I tried version three at one stage but stuck with version two. It was so long ago, I can't actually remember why. Laziness maybe....

If I remember correctly, the issue I mentioned specifically affects version 3.5, I made tests with version 2.82 and the same files were copied flawlessly. But version 2.82 doesn't retain all the timestamps, which is the very reason I started using SI! 3.5 as my go-to copying/synchronizing application (in combination with Robocopy which I discovered later). I can provide the updated/corrected version of SI! 3.5 by private message if needed, but wouldn't release it publictly without the author's consent.

How would I go about creating a sparse file to sync for testing? I'm not sure I've seen one. Are they a new Windows thing or an NTFS thing? Can XP can play too?

Like I said, I had this problem with all files downloaded with JDownloader 2 and FlashGet (probably Orbit Downloader too but it's become a very suspicious program). All on WinXP, and yes it's an NTFS feature, intended to reduce the occupied size for files which have a lot of empty spaces (what are we living for ?). The “sparse” attribute isn't specified in the attributes column, at least in WinXP (I just tested : in Win 7 it does appear with the “P” letter). Fsutils (command line tool included in Windows) can set the “sparse” attribute with this command : fsutil sparse setflag "filename". A particularity of those files is that the “size on disk” and actual size are exactly the same (that's the only other thing I could notice about those files, since I had never heard about that “sparse” thing, maybe that's what put the author on the right track to finding a solution -- actually I reported that issue twice, with a 5 years interval (!), and only the second time, in october 2015, did the guy investigate it thoroughly, the first time he may have thought that the corruption came from my computer). For instance, I had a folder with 24 video files (TV programs from a “replay” service downloaded with JDownloader 2), for a total size of 6 664 707 166 bytes, and the “size on disk” was exactly the same ; if copied with SI! 3.5 those files would retain this characteristic (and hence the “sparse” attribute) but end up corrupted as I explained (those 6GB worth of files could be packed into a 4MB RAR file ! and the copy process was abnormally fast for 6GB, which prompted me to verify in ultra-paranoid mode, but rightfully so) ; if copied with Robocopy, they would lose the “sparse” attribute (Robocopy also removes the “compressed” attribute, possibly others, whereas SI! 3.5 really keeps everything) and the total “size on disk” would be slightly higher as it normally is (when file boundaries don't exactly match sector boundaries, there's some “slack” space at the end of the last sector, which can't be used by another file, hence the discrepancy ; with the “compressed” attribute the “size on disk” is lower on the contrary, except for files that are “resident”, i.e. small enough to be entirely located in the MFT : when set to “compressed” those files get copied to the data area where they occupy a whole sector, and so they occupy more space than when they were uncompressed, thus using NTFS compression on a large number of very small files can actually reduce the free space significantly). (And of course once they had been copied with Robocopy and were no longer “sparse” they could be copied correctly with SI! 3.5.)

Some pages I saved on the subject (I don't remember what they say but if I saved them then I guess they were at least remotely relevant) :
https://en.wikipedia.org/wiki/Sparse_file
http://superuser.com/questions/508801/removing-sparse-file-attribute
http://blogs.msdn.com/b/openspecification/archive/2010/11/05/notes-on-sparse-files-and...e-sharing.aspx
http://www.codeproject.com/Articles/53000/Managing-Sparse-Files-on-Windows
https://technet.microsoft.com/en-ca/library/cc753059.aspx
http://netez.com/bbs/viewtopic.php?t=2750
http://ss64.com/nt/fsutil.html
http://ss64.com/nt/attrib.html > “attrib” is supposed to also deal with “sparse” attribute but it doesn't seem to work on my Win7 system (confirmed in the comments below this article)

[drjtech]

3) Your USB stick may be corrupted, or even counterfeit. There are tools to test them. (H2testw, ChipGenius, USBFlashTools -- I tried those a few years back when after I had bought a counterfeit SD card, but don't remember which does what and how well.)

This was my first thought as well. A counterfeit can appear to be 64GB but actually only be 4GB, for instance.

Lesson learned (hopefully) is to backup your files. Backing up a file in no way means you can safely delete the original.

When I copy files to external media I tend to use QuickPar http://www.quickpar.org.uk/ to generate PAR2 files for integrity checking and repair.

[hello_hello]

abolibibelot,
I haven't read your whole post yet... The real world calls... but I will later.
The only file copying issue I know of in respect to corruption and USB3 is a Win7 issue. I don't recall reading similar complaints about newer Windows.

I've not experienced the problem but I've no doubt it exists. I've read threads on the subject in Microsoft forums and there's one or two threads on the topic in the Computer sub-forum here at VideoHelp.

[hello_hello]

abolibibelot,
Is NTFS file compression and sparse files the same beast with different names? I found a sparse file option for utorrent and enabled it and forced it to create a few new files, but I'm not sure if anything changed. Size on disc and file size weren't the same. If sparse files rely on NTFS file compression that may be the problem. I'm somewhat sure file compression is an option you select (or not) when a drive is formatted and I'm more than somewhat sure I've never enabled it. Would that be why I'm not seeing any sparsness?

I haven't checked much more thoroughly yet, although I only use Explorer to install xplorer2 so I never have to use Explorer again, and I'd be surprised if it didn't show such an attribute, although anything's possible. I'll play around some more in a day or two.

Cheers.

[abolibibelot]

Is NTFS file compression and sparse files the same beast with different names?

Note quite, from what I understand. NTFS compression does compress, don't know exactly how or how it actually appears on the drive (now I know, see below), but the file size is significantly reduced when a file has a lot of redundancy even if there are no empty parts. For instance I created a file with 1000 lines of “herpderpherpderp....” (as if I hadn't more important things to do ! ;^p), actual size is 145998, size on disk is 147456 (= 36 x 4096, or 36 4KB sectors), if set to compressed it gets down to 20480 (5 sectors). If then accessed with an hex editor like WinHex which can open the whole volume (to bypass the normal behaviour of the system which is to uncompress the file on the fly) it appears compressed indeed :



That compression is not very efficient though : if the same file is compressed in ZIP, size gets down to 868 bytes, 288 bytes in RAR (set to “normal”), 264 in 7Z (default settings).

The “sparse” attribute is just supposed to remove blank / empty parts, keeping the rest as is -- yet the “size on disk” always appears to be the same from what I could see so far, even when there are indeed a lot of empty parts, so it's even more mysterious. If I take my herpderp file, replace lines of “herpderp” except the first, the last, and one in the middle, with either spaces or “00” bytes (while keeping the same size, so there should be empty sectors), it appears exactly the same in WinHex if set to “sparse” or not... Maybe the empty parts must be bigger than that, or it doesn't work like that at all, can't say more at this point.

If sparse files rely on NTFS file compression that may be the problem. I'm somewhat sure file compression is an option you select (or not) when a drive is formatted and I'm more than somewhat sure I've never enabled it. Would that be why I'm not seeing any sparsness?

No, that's independant, there can be compressed or sparse files even if the volume hasn't been formatted as “compressed” (and there can probably be non-compressed files on a volume formatted as “compressed”, just like a folder which is set as “compressed” can contain non-compressed files). If you set the “sparse” attribute with the fsutil command you should see that the “size on disk” appears identical to the actual size, and a new “P” attribute (with Windows 7 at least -- just tested with Vista : the “P” doesn't appear).
And then, going back to the origin of this off-topic discussion, if you attempt to copy that file with Synchronize It! v. 3.5, the copy is going to be corrupted after exactly 25000 bytes (not with v. 2.8).

I haven't checked much more thoroughly yet, although I only use Explorer to install xplorer2 so I never have to use Explorer again, and I'd be surprised if it didn't show such an attribute, although anything's possible. I'll play around some more in a day or two.

I'm not sure if I ever tried Xplorer2, apparently it's a commercial program. I was used to Explorer in Win XP but on Win 7 so far it's a nightmare (the search function is almost unusable, and there are many supposed improvements which make formerly simple, intuitive operations unduly complicated to perform) and I'm gonna need an alternative. It's a shame that the central piece of an operating system, the one that should allow the end user to, well, operate things, needs to be replaced to restore a satisfying experience.
Total Commander (8.51a) doesn't show the “P” attribute (but it has a quite antiquated interface, I use it for specific features but couldn't make it an Explorer replacement for regular file operations).

[lg1988]

I really appreciate the help. I am pretty clueless in this area but thanks to the replies on here, I have a few things I can try now. If I have to spend the whole weekend on it, I will.

If anything, it's taught me to be a lot more careful with what I use to back-up my files in future.

Thank you

[hello_hello]

I'm not sure if I ever tried Xplorer2, apparently it's a commercial program. I was used to Explorer in Win XP but on Win 7 so far it's a nightmare (the search function is almost unusable, and there are many supposed improvements which make formerly simple, intuitive operations unduly complicated to perform) and I'm gonna need an alternative. It's a shame that the central piece of an operating system, the one that should allow the end user to, well, operate things, needs to be replaced to restore a satisfying experience.
Total Commander (8.51a) doesn't show the “P” attribute (but it has a quite antiquated interface, I use it for specific features but couldn't make it an Explorer replacement for regular file operations).

There's a free (lite) version of xplorer2. I haven't checked but I thought that was the one I linked to. It doesn't have all the features of the paid version but it's still better than Explorer. I haven't used Total Commander in years but if it doesn't have tabs......

My main preference was for a file manager with dual panes (ie like running 2 instances of Explorer side by side) tabs, and the configuration had to be savable so it opens to the saved state (although it can be set to open to the state it was in when last closed if you prefer). Tabs being very important.
I have it running dual pane, each pane has four tabs, and each tab opens to a frequency used location. Each tab can be configured with different columns displaying different information if required. I save the configuration and tell xplorer2 to always start up that way. Having the equivalent of eight Windows Explorer instances open when moving files around is someone convenient.
Unfortunately the free version has no search function. Unless that's changed as the version I'm using is a year or two old now. I should update it.. but it's not something I use all that much so I can tolerate XP's search, although at one stage I had a third party search utility installed and selecting search in Xplorer2 would open it. I can't remember the details right now or if it's still do-able, but I'll try remember. Explorer right click menus are integrated into xplorer2.
Someone else pointed at another dual paned file manager not so long ago that looked interesting. I think I made a note but haven't got to it yet. I'll try to find it later.

I vaguely remember a discussion about xplorer2 running on Win7 behaving slightly different than on XP. Inheriting a Win7 Explorer annoyance. I can't remember the details but I'll look for that later too.

Thanks for all the sparse file info. I've read it but I'll have to wait a day or two before I can play around again as I've got another couple more long days out in the real world to get through.