Extraction of raw A/V from AVI container for files >4GB?

[alfonzo]

I've got a bunch of high-definition files that were encoded with Dr. Divx. Many of them are larger than 4GB, which is causing no end of problems. After a great deal of thrashing, I've finally learned that this is due to the AVI container that the files are packed in (it's old, it's 32-bit, and it can't handle playback past 4GB, and that's straight from a tech at Divx Labs). Playback stops at the 4GB point, and I need to fix all these large files so that won't happen any more.

The problem isn't the encoded data in the files--it's MPEG4, which doesn't have a large file size issue. The problem is the old AVI container, and how the heck I can get the complete source audio and video files OUT of it and into something more modern (.mp4 or .mov, preferrably).

I'm ripping my hair out on this one. Every tool I've tried on OS X, Windows, command line stuff--demuxers, video script tools, splitters, you name it--politely stops processing at the 4GB point. VLC will spit out a .ts file under the Advanced Output > dump raw file option, but I'm not sure (a) what this file is--it's not a transcode, it was written too quickly, or (b) what I can do with it yet.

Any suggestions? As the pirate with a steering wheel in his pants said, it's driving me nuts.

[jagabo]

The original AVI spec only allowed 4GB files. Most programs now use the ODML extensions that allows for bigger AVI files.

If you are using a hard drive formatted with a version of the FAT file system you will be limited to 4GB files -- this is a FAT file system limitation.

If you are burning to DVD the ISO format only allows 2GB. The UDF file system allows up to 4GB.

.TS is an MPEG transport stream file.

Try using VirtualDub in Direct Stream Copy mode to cut large files into smaller pieces.

[alfonzo]

Thanks. I'm aware of the FAT32 limitation, not an issue in my case. I don't have any drives formatted with FAT32. I'm not burning any of these files to optical media, they're all staying on hard drives.

I did try VirtualDub (and NanDub), both stop writing at the 4GB point in the large files, regardless of the clip size I select.

Matroska (mkvmerge) is running now. We'll see how it does...

[alfonzo]

And for the record, I know what a .ts file is, I just don't know what the heck VLC did other than (I'm guessing) copy the .avi into a new file and name it .ts for no apparent reason. There's no way it could have transcoded the movie from MPEG4 to MPEG2 that quickly.

[jagabo]

I handle AVI files larger than 4 GB all the time with VirtualDub. Sometimes over 100 GB. I suspect Dr. Divx screwed up when it made the files so you may never be able to recover the data. Can you play the files all the way to the end?

[alfonzo]

Entirely possible.

What processor are you running on for those big files? Any chance it's a 64-bit chip?

[jagabo]

I using a Core 2 Duo which is a 64 bit chip but since I'm running Windows XP 32bit it's not really functioning as a 64 bit processor. A 32 bit chip can read >4 GB files with no problems -- as long as the program is written correctly.

[alfonzo]

Hmm. Thanks.

The data appears to be there, just judging by the file sizes. There's gotta be a way to extract it...

[alfonzo]

I wonder if there's a setting in the AVI header that's causing all the tools to think the files are truncated.

Guess I've got some more reading to do...

[jagabo]

Here's the ODML spec.

http://www.the-labs.com/Video/odmlff2-avidef.pdf

Note particularly the part about files greater than 4GB. Your files were probably written with a program that didn't implement the ODML extensions but kept writing data to the end of the file. So the RIFF structure is corrupt. In theory it may be possible to extract the data and repackage it but I don't know of any programs that will do that. You might try DivFix or VirtualDub's "Scan Stream For Errors" but I don't think either of those will work.

[alfonzo]

I don't think so, either. Looking at the file info from AVI-Mux GUI, I suspect the header is telling every app that frames stop at the 4GB point, even though as you surmised the encoder apparently kept running to the true end of the source video.

[vhelp]

I recently upgrade from WIN98 (FAT32) to XP Home (NTSF) but I've been dealing
with the 4 gig limit and getting by, all these years.

You might also try AVIsynth, using either the AVIsource() or else DirectShowSource()
functions to open he file. Then, feed (frameserve) that into virtualdub and do
a *new* direct stream copy (or another codec/compression container) instead.

The thought process behind this sceam is to see if these methods for opening
AVI files will use a better algorithem. But, since these will be feeding one
image at a time, it might allow access past the 4 gig limit of the original
sources container. But I guess it all depends on how AVIsynth (internally)
will handle your source file. It won't hurt to try, however. Using Notepad.exe,
type in the text you see in blue and save.

NOTEPAD: script.avs

x = "path\myfilename"
AVISource( x )


..or..


NOTEPAD: script.avs

x = "path\myfilename"
directShowSource( x )


Then, open virtualDub and drag the script.avs file into the window, and see how
far you can move inside the time line.

-vhelp 4282

[alfonzo]

Thanks, vhelp.

AVISource didn't work, it doesn't like the AC3 audio. directShowSource politely goes to the same old 4GB end point. But I'll keep at it...

[vhelp]

I don't know how badly you need these video files. But if you do need
them, then keep trying.

Here's another idea to try.

Using TMPGenc: Option\Environmental Settings..\VFAPI Plug-in
try setting the priority level of the following, to see which one
*might* open completely, the 4 gig file size. You do this with
File\Preview\ and see if you can go past the mark.

bump these around (priority'wise) in hopes that it will open the
file in the mode listed below. You right-click the file and select
the priority level. Keep doing it to the one that you want to bump
up to the top.

[x] AVI2(OpenDML) File Reader
[x] AVI VFW compatibility Reader
[x] DirectShow Multimedia File Reader

-vhelp 4283

[alfonzo]

Okay, now we're getting somewhere.

The AVI2(OpenDML) setting does bring up the correct full file length (2:20 instead of 1:52, if anybody's interested) in Preview. Now, how can I use this? I'm not interested in re-encoding to MPEG2, I'd really just like to strip out the source A/V files and remux in a better container. Anybody know of a resource similar to TMPGenc that might do that?

[vhelp]

alfanzo, to remux, all you have to do (now) is strip away the
audio (or save as wav or something like that) and then (if memory
serves me) you bring both video and audio into virtualDub and process
it. This is something that I don't do in my projects because I don't
have these types of video's. anyway.

But, I think you will have to re-encode your (*.mp4) AVI to another
format because nothing seems to open them, except TMPGenc.

At least you can get the audio (below) or you can re-encode, from scratch,
to something else.

Getting the audio:

While still inside TMPGenc, do the following to get the audio portion:

--> File\Output to file..\wave file..\

You can do the same for the video (above) but select, AVI, and then choose
another codec and let it save it as such. Or, re-encode to MPEG

Perhaps someone else can help you further.

-vhelp 4284

[jagabo]

You can view the RIFF structure with VirtualDub. Start VirtualDUb then select Tools -> Hex Editor. Open a file from the Hex Editor with File -> Open. After opening the file select Edit -> Riff Chunk Tree. That will show you the RIFF structure (it will take a while to parse the file).

[alfonzo]

Thanks, guys (assuming that you're, er, guys . I've got an AVIMaster run going right now, I'll report back on the results. The GUI version caught errors in the RIFFs, most obviously the two different file lengths/frame counts. Not sure whether the repair tool will work or not, but at least I know more than I did this morning.

[alfonzo]

Hmm. Well, I think I'm making progress.

Based on the RIFF dump from AVIMaster, it appears that Dr. Divx writes the frame count up to the 4GB limit in the AVI Main Header (for this example, that was 161626 frames, or 2775A Hex; 5A 77 02 in the Hex editor). AVIMaster tells you this is an error, since there are more frames than that in the actual MPEG4 stream, but it won't fix the problem.

So what the heck, I tried brute force. (I do have an unchanged safety copy in case this blows up in my face).

Using TMPGenc, I determined the actual number of frames as 204328 (shift and left-click on the scroll bar at the bottom of the Preview screen to get the last frame), Hex 31E28 or 28 1E 03 in Hex editor. I opened the file in VirtualDub's Hex editor and changed the appropriate values (the frame count is near the top of the Hex dump), then saved the file.

I re-opened the file in VirtualDub first, and no change. Then I tried again with extended option of AVIFile compatability mode, and the full length was there in the search bar. Figuring it would be a good idea, I re-opened again and also selected the re-keying option. A couple of hours later, the file was open and presumably re-keyed (this may have been a mistake--see below).

Then I tried to save as a pass-through, and the save stopped at 2GB! Same behavior with file splitting. Somehow, I am reverting through even worse versions of the AVI spec... but I seriously do think I'm making progress. I'm running a frame error check now, and will report back. If it fails again, I'll repeat all of the above except the re-keying option and try to save again.

[alfonzo]

Aw, crap. No joy. VHub can open anything (modded in Hex editor or not) with the AVIFile Compatibility Mode, but writing out a new version stops at 2GB every time. Splitting doesn't work, either. Now I'm stumped again.

[alfonzo]

Whew.

Well, it really took a long time to figure this out, but I think I have a fix. Standard YMMV disclaimers apply, but if anybody else runs into the 4GB limit thanks to Dr. Divx, here's how I fixed it:

Run the affected file through AVIFixed (I used 1.1b1). That'll strip the old, faulty header off the .avi. Then open the result with VirtualDub, which will rebuild the header when it loads (this takes a while). Save another copy with VirtualDub using Direct Stream Copy for both streams.

In Graphedit, use the 3ivx 5.0 (4.5 will not work) codecs to convert the AC3 audio (if that's what you have) to AAC and remux to .mp4 or .mov (I prefer .mov, YMMV). The working graph for the video side looks like this:

[source.avi] > [AVI Splitter] > (video) > [3ivx Media Muxer] > [filewriter]

... and for the audio track coming out of AVI Splitter:

[AVI Splitter > (audio) > [AC3Filter] > [3ivx Audio Encoder] > [3ivx Media Muxer] > [filewriter] (same muxer and filewriter as above, natch)

The result plays back fine on either a Windows PC or a Mac, assuming the 3ivx codecs are installed.

Thanks to everybody here for the suggestions and advice. I should note in closing that Dr. Divx still works great for smaller files (i.e. hour-long TV episodes), but I wouldn't trust it for anything longer than 1:45 or so.

[alfonzo]

A bit of an update here... since 3ivx 5.0 won't connect to a 5.1 audio source without buying the ($99) Pro version, and because the 3ivx 5.0 codecs interfere negatively with so many other codecs on the Mac side, I've had to find another method. Fortunately for me, Perian 1.0 came to the rescue. Here's my current conversion process:

1. Run the long .avi file through AVIFixed to strip the header.
2. Open resulting file with VirtualDubMod, then save a new copy.
3. Demux the .ac3 stream from the new copy with VDM.
4. Open the new .avi and .ac3 files with an MKV muxer and build an MKV file of the full movie. If you're happy with using .mkv-friendly playback tools, you can stop here. Since I use FrontRow at my HTPC (er, Mac), and since Quicktime doesn't do MKV without some help, I needed another step.
5. On a Mac, load Perian 1.0, then open the .mkv file in Quicktime Player (you'll have to choose "Open With" and then select "All Applications" to get to QT Player as an option).
6. Allow the full movie to load (watch the status bar at the bottom of the player window). Once the movie is loaded, Save As to a reference movie. Do NOT delete the .mkv file!

Quicktime will now open and play the full movie from the small reference .mov. This is quite nice, since you now don't have to recode the .ac3 audio.

[alfonzo]

A bit of an update (and hopefully the last one): if you have the patience for still one more demux-and-remux step, once you have a working .mkv file from the procedure above, you can demux THAT file using MKVE Wizard, then remux the resulting "laundered" components with VirtualDubMod into a regular old .avi that actually works. Gives you the advantage of portability if that's important to you (it is to me), since the Perian MKV fix for Quicktime/FrontRow is not just ungainly but also not entirely dependable.