How to scan for short .mpeg fragments and join them together to a movie

[camaro76]

I had a Panasonic DVD/HDD where I had many movies recorded. In an attempt to transfer the videos to a second hard drive I hooked the drive up to my computer and tried to browse for any video file. But I could not see any files at all, maybe due to a non-supported file system or problem with the index header. After I inserted the drive back into the Panasonic recorder I was not able to see any files at all. So I assume that Windows did alter the index or something. Then I removed the drive from the Panasonic recorder and hooked it up to the computer again, and used a third-party program to extract what was possible from the drive. And I think I got all the data that was on the drive.
Now, the problem described..: Even if I have all the data, ie. all the mpeg segments on a external hard drive, all the movies I had on the 250Gb drive are combined divided into more that 800.000 file segments, and their filename are not numbered in sequence. I can find 10 or even up to 30 segments with filenames in a row that belong together, but then I have to browse through the whole disk to find the next few segments and copy them as well to a new folder. And every time I scroll the windows just a tiny bit, since there are almost a million files, it takes some time before the thumbnails shows up in the window. Quite a pain in the ...
Each mpeg segment is only a fraction of a second long, and to patch a whole movie together (and there are many movies on that drive) will take days and even weeks.

What I need is a program that can recognize / analyze file segments and put them in the right order, regardless of the filename
Of course the change from one scene in a movie to a new scene may be a problem, since there will not be any way the software can know what will be the next scene, but at least if all the video segments that belong in the same scene would be sorted in a row, I can manually organize the order of scenes.

I assume there must be some kind of software that can analyze short video segments and see which one is next in line, but I have failed to find one, even after many hours on the Internet.

I have purchased programs like CnW Recovery and many other programs that are supposed to do forensic analyzing and so on, but absolutely no program has been able to even remotely do the job, but rather tend to list the file segments I already have, but under a new name.
Is there such a program out there, and what is the name of it (or names in case more than one piece of software are up to it...)?

To make the scenario even worse, now and then one of the file segments (even if they are in a row regarding filename) has another file name extension then.mpeg and has a length of 0.000 second (but it can still play as the other files that has a .mpeg file, even if it's supposed to be only 0.000 second long, so it is not zero length...).
Due to this I can not use a regular joiner program to stitch the video segments I have found together. That is, I have tried many programs but all of them have failed to join the normal .mpeg files and the other zero-length files together.

Any suggestions?

[hech54]

Ouch.

[DB83]

I doubt any such program exists.

My guess is that the program you used to transfer the files merely copied the raw sectors from the drive without any reference to the File Allocation Table. If you have individual files you might attempt to simply join them together and re-edit the large file. By no means an ideal method.

Windows should not have done anything to the drive when you first took it out. More likely that the Panasonic thought it was a new drive.

Hindsight is a wonderful thing and had you asked before you did anything there were simple things you could have done straight from the Panasonic. There is also software that, possibly, could have read the file system from the HDD (usually based on Unix). Sorry I forget the program's name right now but you could always Google.

[camaro76]

Yes, I guess that the FAT table got replaced. I know there are software that actually compare the content of a file, I have such programs, but they are useless in this case. These programs actually compare the file itself to check for duplicate files, regardless of filename and location. But maybe there are no software that detect gradually changes from one mpeg snippet to the next, as I need to do.

[DB83]

No. Not replaced. You would have had to format that HDD to do that. Windows can not see the files due to the proprietory format. And, as I said, the DVR thinks it is a new drive.

Unless you can find a program to read it then you are stuck with these cluster snippets. The problem is that your recordings are scattered across the drive just like any HDD. So the earlier sectors could be any part of the recording. Essentially impossible to recover unless the FAT can be read.

[abolibibelot]

How did you get those 800000 file fragments ? What is the pattern of their filenames ?
If you still have the original HDD untouched (except for that messed-up index header or whatever), a tool called PanHDD might be able to recover whole videos from it. I also remember reading a thread where a guy had managed to decipher the filesystem on a Panasonic HDD recorder, and had written a script to recover the files from a Windows PC, I must have kept that somewhere... ah, here it is !
Photorec might be worth trying too. But MPEG/VOB video files are especially difficult to recover by raw file carving, as they are made of small segments, each having a header that could be the begining of a file, as opposed to other video formats which have a specific header at the begining ("RIFF" for AVI, "ftyp" for MP4...).

[Cornucopia]

That's the problem - it's no longer untouched.

Mpeg ps & ts do usually have time stamps (presentation order & display order) with which, if you can parse for them, could assist you in getting things in better order.

For others who haven't done something like this yet but are considering it, I would suggest you never mount a recording settop's drive in a PC. Clone the drive first, and then put it back immediately, then do the work ONLY on the clone.

Scott

[abolibibelot]

That's the problem - it's no longer untouched.

Here I meant "untouched" as in : were the actual storage areas (as opposed to filesystem / metadata) not overwritten ? (by reformatting it inside the recorder and recording new videos).
I haven't tried the tools I suggested in a long time (the ones specific to standalone Panasonic recorders), so I don't remember if they require the filesystem to be in a valid state to do their magic. I think that PanHDD has an option to extract a whole image from a drive and then work from that image.

Depending on the software used to get those thousands of fragments, there would be a possibility that many of them are contained within bigger ones, thus being useless duplicates. I've had a case like this using R-Studio, it recovered for instance 50 files from the same spot of the same movie, each one being (IIRC) 2KB smaller than the former – that would make such an issue all the more tedious to sort out...

If the original HDD is no longer available or has indeed been overwritten, there might be a way to paste all those files into a single large file, using a clever batch command, and then open this large file either as a volume image file with a decent data recovery software that can efficiently extract MPG files by raw carving method (Photorec which I already mentioned is pretty efficient, recent versions of R-Studio have improved in that regard), or try to open it with a MPG cutter which doesn't transcode and lets you make precise edits, like DVBCut.

– What about the contamination?
– It's all contaminated.
Breaking bad, “The fly”

[pandy]

Panasonic frequently use special index file to describe pieces - usually (i may wrongly recall something i saw over 8 years ago) it was 3 files, one with audio video (normal TS), side to this description file (some form of EPG) and special index file to speedup trick modes - i would focus on those two files - analysing them you may have more possibilities to detect and combine this puzzle box.

btw
side to PTS and DTS there is also GOP time stamp http://dvd.sourceforge.net/dvdinfo/mpeghdrs.html#gop and it may be the most usable hint

forget to add that HDD can use own file system and Windows may incorrectly recognize such file system - commonly allocation size on such file system is 188 bytes, sometimes it is 204 bytes so sector is commonly multiple of 188 (204) dividable by computer sector size i.e. 512.