VideoHelp Forum

Try DVDFab and copy Ultra HD Blu-rays and DVDs! Or rip iTunes movies and music! Download free trial !
+ Reply to Thread
Results 1 to 1 of 1
Thread
  1. It's not strictly video related, more computer related, so I'll put that here.
    It's a brief report of my experiences with data recovery of video files, with the so-called “raw recovery” method, also called “file carving” or “recovery by file signature search”, which can still be attempted when the more efficient approach, through the analysis of the filesystem structures, fails to retrieve a specific file, or anything at all (when either the filesystem has been damaged by formatting / accidental O.S. install / corruption / God's wrath / whatnot, or when the specific metadata pertaining to the file in question was wiped following deletion). I'll mainly mention three data recovery utilities I know quite well : R-Studio (a full fledged commercial data recovery software), Recuva (a freeware, which is generally less efficient than R-Studio at reconstructing damaged filesystems, but is still impressive despite its stripped interface, outperforming R-Studio in at least two aspects : assessing each file's odds of successful recovery, and “raw” recovery of AVI files — see below) and Photorec (also a freeware, a dedicated “raw file carving” program which has been favorably compared with very expensive professional forensic softwares, although it is not without flaws). Note / disclaimer : most of the links in this post point to threads I created on other forums, so the corresponding posts also relate to my own experiences.

    AVI, WMV, MP4, MKV files are generally very well detected and recovered so long as they are contiguous (not fragmented). R-Studio and Photorec recognize all those file types ; Recuva doesn't detect MKV (it hasn't been updated since 2016), but is about as efficient as the other two with AVI / WMV / MP4. R-Studio has a lingering issue with AVI files, though : in some cases, two distinct complete files are merged as one ; I made a detailed report of the issue on the dedicated forum, with no feedback whatsoever so far.
    FLV and MOV tend to produce many false positives (R-Studio, Photorec, Recuva), while real files of either type are usually recovered well.
    MPG / VOB and MTS seem to be particularly tricky to properly recover, because of their pattern : header structures (and therefore file “signatures”) are found at regular intervals within a single file, which can trick most DR softwares into considering that some or all of those headers are the beginning of a new file. R-Studio is a major offender here, as it can detect hundreds of “extra found” files for what used to be a single MPG or VOB or MTS file. Photorec is much better at dealing with those file types but can have the opposite issue, as it usually detects VOB files quite well but can merge two of them (it can also split them into several files, but not to such an extreme extent as R-Studio).

    Another issue is that video files which are complete and contiguous and would normally be fully recoverable can be interrupted by the detection of a random false signature of another file type within the data stream (or sometimes a legitimate file embedded in the header, as is frequent with MKV files). It happens a lot with Photorec, which by design stops the recovery of file N whenever it detects what it considers as a header of file N+1 (or sometimes attempts to find complimentary fragments of file N elsewhere on the volume, but almost always fails — see below), and which also tends to detect a lot of false positives within valid video files, especially for JPG and MP3 file types (I've seen perfectly fine video files recovered by Photorec as butchered blobs riddled with holes along with dozens of ~3KB dummy MP3 files, or truncated beyond a dummy ~30KB JPG file). As a result, the recovered video file is truncated and generally unreadable beyond the first cutting point, or completely unreadable if playback relies on an index at the end (which is the case for some MP4 files which have their “moov atom” section at the end — partial AVI files can still be read despite having an index at the end, in which case for instance VLC Media Player proposes to reconstruct the missing header before playing it). R-Studio and Recuva can also find a false JPG (or other) signature inside a video file, but still detect / extract the complete file, if it is not fragmented.

    If a file is fragmented, it gets really tricky to fully recover it. R-Studio or Recuva don't attempt to reconnect incomplete fragments ; R-Studio generally cuts the detected fragmented file as it detects the header of another file (it can still add an extra segment if no other file was detected between the end of the valid first fragment of file N and the beginning of file N+1, in case either there was no other file in-between, or there was a file of a type of which the search was disabled in the options — if it's clear enough) ; Recuva seems to happily extract a file with its full size (if it's indicated in the file's header, as is the case for AVI and WMV at least), even if it overlaps with other files it detected during the same scan. Photorec, on the other hand, usually attempts to find the next fragment(s), but almost always fails, in anything other than the most simple scenario where the next fragment is located right after a file or group of files in between — it may work sometimes for a memory card used in a camera, on which files are recorded mostly sequentially and fragmentation occurs only whenever a few files are deleted, but it's considerably more complex on a multi-terabytes HDD where files' pieces can get written just about anywhere (on a 3TB HDD I've thoroughly analysed, which had generally very little fragmentation, I've still found files which had fragments located hundreds of GB apart, sometimes fragment N+1 can even be located hundreds of GB before the beginning of the file — for instance, there was an incomplete 4GB MKV file around the 1585GB mark, I managed to manually find the missing 370MB fragment around the 116GB mark, while Photorec pasted an erroneous fragment found around the 1591GB mark), making it a nigh impossible task to reconstruct them with an all-purpose data recovery software. Only very advanced “file carving” utilities claim to successfully reassemble fragmented files ; I haven't tried those so couldn't comment on how efficient they are. But in case of massive fragmentation, which can happen even with plenty of available space when downloading several files simultaneously with a software that doesn't preallocate each file's full size, I highly doubt that even that kind of utility could successfully reconstruct a single file (which I managed to do, as exposed in the above linked thread, with files which each had 6000-12000 fragments, only because I was wise enough to save a complete list of each file's clusters, obtained through various means{1}, before the source HDD started to seriously misbehave — although also not-so-wise to fail to save the complete MFT while it was still recoverable).

    It is also worth mentioning that on the NTFS filesystem, when a file larger than 4GB is deleted, the allocation information in its MFT record seems to get systematically wiped right away{2}, by design, meaning that it's no longer possible to find its precise location on the partition, or to retrieve all its fragments if it was fragmented. But its size is still present ; in such case, R-Studio always reports a “0” size ; Recuva generally indicates the correct original size (unless the file had an “alternate data stream”, in which case it indicates the size of the ADS, for instance a 4.3GB ISO file is reported as “26 bytes” because it had a 26 bytes ADS, which hasn't been deleted as it's “resident”, meaning, embedded within the MFT record) along with a comment saying “File's data could not be found on the disk” ; while DMDE, another full-fledged data recovery software (among the three most often recommended at forum.hddguru.com, with R-Studio, and UFS Explorer which I haven't tried — I haven't as much experience with DMDE as with the other tools mentioned here when it comes to “raw” recovery of video files so I can't comment on its strengths and weaknesses), allows to directly examine the file's MFT record and display its original size even if the file allocation information has been wiped (and even if it had an ADS). Knowing the correct size, it can be verified if the such a large file was properly identified by the “file signature search” approach, by searching if there's a file with that specific size among the “extra found” files of the relevant type. If not, it can mean that the file was fragmented, or that it was interrupted prematurely by the software because of a false positive. In this case, knowing the correct file size, and where it begins, it's easy to manually extract it with a hexadecimal editor (with WinHex for instance : Tools > Open disk > [select correct device / volume] ; Navigation > Go to sector > [type the sector number indicated by R-Studio's hex viewer in the “Properties” tab, or indicated in the file name generated by Photorec : f123456.avi = file beginning at sector n°123456] ; right-click > Beginning of block ; Navigation > Go to offset > [type the file's size displayed by Recuva or retrieved by examining the MFT record with DMDE] ; right-click > End of block ; Edit > Copy block > Into new file [choose location and file name]). As this can be tedious to do for more than a few files, another option is to run a new scan after disabling the file types which tend to produce false positives, or, to be on the safe side, to enable only the file types which produced truncated files through the initial scan (even then there can be issues, as false MPG signatures can be found within other video file types for instance, and it's also tedious to have to run multiple scans to get an optimal recovery from a single storage device — but hey, that's why backups are important).

    It might be useful to someone, somewhere, some day...


    {1} Including Piriform's Recuva and Defraggler, which might be the most efficient and straightforward combo (at least when it comes to free utilities) to get that kind of information : Defraggler can indicate which files are located within a given “block” (unfortunately it's not possible to request a specific interval of sectors / offsets — I sent a message to the developpers some years ago, suggesting that this would be easy to implement at this point, making it a killer extra feature), while Recuva displays the complete list of a file's clusters in the “Info” tab. (R-Studio likewise provides a list of sectors but doesn't allow to copy or export it in its entirety, only one value at a time can be copied with CTRL+C which is totally impractical — I reported this issue repeatedly but it hasn't been improved so far.)

    {2} It also seems to be the case for highly fragmented files which have several MFT records pertaining to them, when the $DATA field is too large to be “resident” within a single 1024 bytes record. It can also happen for smaller and contiguous files, for no obvious reason.
    Last edited by abolibibelot; 13th Oct 2020 at 03:23.
    Quote Quote  



Similar Threads