Archiving Video - Smaller than DV better than MPEG-2?

[boblin2]

I have many hours of video in DV format. The most important video has been archived to DVD (200 DVDs) and hard disk (750G) in DV format.

I have other video that is not as important to me. What CODEC should I use to archive this material? The ideal CODEC would give better visual results than MPEG-2, but will have the same or smaller file size than MPEG-2.

For future authoring, I want to keep the characteristics of the original video i.e. interlaced PAL at 720x576 and interlaced NTSC at 720x480.

Thanks.

[jagabo]

MPEG 4? ASP or AVC.

[boblin2]

jagabo: 2 seconds to read my post and post a reply!!!

I'll give MPEG-4 a try. Thanks

[boblin2]

DV to MPEG 4 Conversion (using SUPER C v2008 build 30, auto selects ffmpeg encoder).

TEST 1
Output Container: mp4
Output Codec: H.264/AVC
Output Audio: AAC
Result: Cannot select interlaced option, only progressive. Video looks fine, but is not interlaced.

TEST 2
Output Container: mp4
Output Codec: MPEG-4
Output Audio: AAC
Result: Video is good for watching, but Vegas and Procoder cannot load it!

Are the above results normal?

[boblin2]

I ran Test 2 again, but this time with the interlace option set to "Deinterlace To Progressive". Vegas and Procoder could now load the resulting video file.

So, conversion of DV to Progressive MPEG-4 works fine, and I can edit the resulting video file in Vegas. When the MPEG-4 file is interlaced Vegas and Procoder cannot read it. How do I fix this?

Also, can H.264 be interlaced as well as progressive?

[NoBuddy]

Why don´t you give Vegas a try?
Load your DV file and encode as Mainconcept AVC custom.
You can adjust framesize, bitrate etc. And interlaced (even what field first !)

Maybe the result comes closer to what you pretend.
Good luck.

[SatStorm]

You can use virtualdub to convert DV to H264 using the modified vfw x264 encoder. You can keep the interlace picture that way and it is fast. The result is on an AVI container and you can choose the audio you like. The file is playable on PCs and later you can use it as a source for other conversions.

I prefer that way myself, than using SUPER encoder. SUPER is good for batch x264 encoding.

[boblin2]

NoBuddy:
I tried the MainConcept AVC as suggested, but again, with interlace (TFF or BFF) selected, the output file was garbled and could not be loaded into Vegas. Thanks anyway.

SatStorm:
Using VirtualDub and selecting 'x264vfw - H.264/MPEG-4 AVC codec' for compression, with the interlaced option checked, it worked. The conversion was slow (I had quality set at maximum i.e. RateFactor = 1.0), but the output video played fine, and could be loaded into Vegas. I converted this file back to DV, and confirmed in VirtualDub, using the ViewFields filter, that it was still interlaced, and it was.

Assuming the codec I selected was the codec you suggested I use, what settings have you found give the results you want?

Many thanks.

[jagabo]

.AVI and .MP4 are containers. The way data is organized within them is different. Some programs won't be able to open the files if you change the extension.

High compression codecs like x264 and Xvid aren't very edit friendly. One thing you can do to improve on this is to reduce the max keyframe interval. Try somewhere around 50.

x264 and B frames in an AVI conta+iner aren't handled very well by many editors. I recommend you turn off B frames.

I tend to use x264 and divx/xvid in single pass constant quantizer mode. Select a quantizer that gives you the quality you want and encode in a single pass. Every frame will get that quality level. This way you always get the quality you want. File size will vary though.

Interlaced mode in the version of x264vfw I have installed (the most recent one from Death the Sheep) doesn't work quite right. The chroma channels are handled incorrectly. You should check yours carefully. The command line version of x264 handled interlace properly.

AviDemux is probably a better choice if you are going to use h.264 encoding. It can produce MP4 files which are better suited to AVC+AAC.

[NoBuddy]

boblin2:
"I tried the MainConcept AVC as suggested, but again, with interlace (TFF or BFF) selected, the output file was garbled and could not be loaded into Vegas. Thanks anyway."

Strange... I just tried it out with a mpeg2 file, BFF.
Using the same resolution (720x480), and had to manually change AR (only offers square and 1.333 as default. So I typed in 0.9091 and rendered as CBR 10Mb/s for a test.
Resulting file was interlaced, BFF and with extension mp4.
Vegas 7 recognized it fine, opened it without problem.
And it showed up exactly as original.
So, for me it´s strange that you can´t open your mp4 file created in Vegas.

[rumplestiltskin]

Try MPEG Streamclip using the H264/MPEG4 option. Play with the parameters.

[guns1inger]

For archiving, DV is the best option. Keep your tapes safe, and transfer as necessary. If you want to be using this video down the track for editing anf converting to other formats, DV is by far the best option open to you.

For playback storage, the advice above is sound.

[PuzZLeR]

You can use virtualdub to convert DV to H264 using the modified vfw x264 encoder. You can keep the interlace picture that way and it is fast. The result is on an AVI container and you can choose the audio you like. The file is playable on PCs and later you can use it as a source for other conversions.

H.264 in an AVI container is an abomination. If you're using low ref frame/low b frame counts, no b-pyramids, then maybe. The aging AVI container wasn't designed to "see" as far as the ranges implemented upon the frames within H.264 and you will definitely lose something if stifling the stream into an AVI. I know VFW is convenient, but it's dying anyway and you really should use MKV/MP4.

Boblin2: If you must compress DV to a form that is lower bitrate, but better quality and lower file size than MPEG-2, then H.264/AVC is indeed your answer. Personally, I'd wait just a few more months now that better and better apps are arriving with blu-ray presets for this codec which would make it even more "standard". (Still looking for a good blu-ray SD H.264 encoder for my older stuff...)

[filmboss80]

Is the cart being put before the horse, Boblin2? If you want to edit the video in Vegas, why not capture as straight DV-AVI? Then, AFTER editing, you can compress the finished product down to MPEG-4 or whatever. Nevertheless, if your original footage is DV, why not archive to mini-DV or DV tape? The cassettes are tiny these days. I come from a background of storing camera-original footage on 1-inch wide tape on giant reels, as well as 3/4" U-matic and Betacam cassettes. Let me tell you, storage on those tiny mini-DV tapes is a breeze, compared to what it once was. I'd hate to see you compress the daylights out of your DV originals, especially if you want to edit the footage.

The question is, how precious is that DV footage to you?

[JimJohnD]

guns1inger and filmboss80 are absolutely correct! You can do copies from tape to tape very easy too. I like to go through my tapes and trim and delete segments and then collect all of the 'usable' footage and send it back to a final tape for safe-keeping. If the footage is really worth keeping, the price of tapes should not be the deciding factor. Buy a cheap camera as a playback deck and keep it with your tapes and you should be good for the foreseeable future.

[PuzZLeR]

I'm not disagreeing with the last two posts myself. If the memories are precious then stick with DV, and the smaller storage medium today is indeed a bonus - as well as being much easier to edit than compressed formats like H.264 and even MPEG-2.

That's exactly what Boblin2 is doing (with most of the important valuable stuff at least).

However, I was addressing another point in the first post.

I have other video that is not as important to me. What CODEC should I use to archive this material? The ideal CODEC would give better visual results than MPEG-2, but will have the same or smaller file size than MPEG-2.

This is why I recommended H.264 in a blu-ray preset for this content - good quality, good compression and future-proofed within a disc standard. This was in case a good compression scheme was desired and after the edits are complete.

[boblin2]

See my post of Apr 12, 2008 00:52 for the settings that I finally settled on.

Thank you to everyone for the input and advice. I have achieved exactly what I wanted using x264vfw and VirtualDub. [The following has been edited a little based on the feedback in the following two posts.]

Job Control was used in VirtualDub to run two-pass encoding of six one-hour videos. With 'Thread' = 1, each pass took 5.5 hours (approximately 5fps), with 'Threads' = 4 on a PC with a 2.4GHz quad processor, each pass took approximately 1.5 hours.

A target bitrate of 6,000Kbit/s produces a video file that is close in size to that of a DVD quality MPEG-2 file. 3,200Kbit/s gives very good visual results. I have decided to use 4,500Kbit/s, with no compression of the audio.

Just to respond to some of the comments:
- The video that I converted to H264 was transferred from 'out of print' PAL VHS tapes bought on eBay, and now we can watch these old favorites here in NTSC land. Any future editing will be limited to creating a compilation of our favorite bits.

- All of our family movies, which were transferred from PAL and NTSC VHS and Hi 8 tapes, were converted to DV format, retaining the original PAL/NTSC characteristics. This gives me the best master to go back to if needed. Original tapes have been stored in a safe place.

- I have backed up my DV format files to DVD (twice) and also to several external hard drives. During the editing process, using Sony Vegas, I added markers which I use to create chapters when authoring the final DVD's. Because my DV is PAL, NTSC, and has these markers I chose not to back it up to DV tape - I also don't have a DV tape device of any kind.

Here's the settings that I used:
(x264vfw build 11_808bm_12665 : http://sourceforge.net/projects/x264vfw/)






Notes:
- The target bitrate is typically set at around 700Kbit/s, and Max IDR-frame interval at 10 times the frame rate (250 for PAL, and 300 for NTSC). The values used above improve quality and future editability of the resulting video file.

- SAR (Sample/Pixel Aspect Ratio) = 16:15 for 4:3 PAL, and 9:10 for 4:3 NTSC.
ProCoder indicates PAL DV has a PAR of 16:15 (1.067), and Vegas indicates PAL DV PAR is 1.0926 (12:11). Putting aside this difference, and using 16:15 for PAL worked fine. Using 12:11 resulted in the video being scaled incorrectly on the ATI Media Player.

- If 'Max consecutive' is not equal to 0 AND 'Threads' is not equal to 1, THEN 'VD hack' has to be checked ELSE the video will lag the audio. That was the case on my machine anyway.

[guns1inger]

Increasing the number of threads to four will allow each core to do it's share of the work. It should not introduce sync problems - it certainly doesn't on my quad core.

The other issue with mpeg-2 and mpeg-4 when it comes to editing is that neither like to be re-encoded, and very quickly show up artifacts. It isn't an issue with straight assemble edits, but if you start doing any image processing, adding titles or transitions etc, you will quickly find the quality is not what you started with.

[PuzZLeR]

Just to add to Gunslinger's comments: For simple edits there do exist dedicated MPEG-2 editors like TMPGEnc MPEG Editor, VideoRedo and Womble that only re-encode the few frames where you did your edits, and Womble can do the more complex stuff like transitions, fades, etc. with minimal re-encoding. I would bet we are not far off down the road from having similar editors for H.264 that work this way too.

Regardless, if you ever must re-encode H.264, you will lose quality, but you would do good using CRF=18. This is considered "near-lossless" encoding.

[boblin2]

Increasing the number of threads to four will allow each core to do it's share of the work. It should not introduce sync problems - it certainly doesn't on my quad core.

The problem was caused by the setting of B-frames Max consecutive to anything greater than the default of zero.

This is my note now on this audio/video sync problem:
- If 'Max consecutive' is not equal to 0 AND 'Threads' is not equal to 1, THEN 'VD hack' has to be checked ELSE the video will lag the audio. That was the case on my machine anyway.

[boblin2]

Regardless, if you ever must re-encode H.264, you will lose quality, but you would do good using CRF=18. This is considered "near-lossless" encoding.

I found that using a Target Bitrate of 3,200Kbit/s produced a file of a similar size to using CRF = 18. Probably not an accurate comparison when it comes to comparing the quality, but good enough for me to have the confidence to lower my Target Bitrate in the two-pass encoding from 6,000Kbit/s to 4,500Kbit/s.

[jagabo]

I found that using a Target Bitrate of 3,200Kbit/s produced a file of a similar size to using CRF = 18.

Maybe for the file you tested. But CRF will always deliver the same quality (relative to the source) regardless of frame rate, frame size, and nature of the video. Multipass VBR will not. With CRF and CQP you specify the quality and the file turns out to whatever size is necessary to give that quality. In multipass VBR you specify the file size and the video turns out whatever quality is necessary to achieve that size.

If 'Max consecutive' is not equal to 0 AND 'Threads' is not equal to 1, THEN 'VD hack' has to be checked ELSE the video will lag the audio. That was the case on my machine anyway.

How much sync error were you seeing? I've never noticed this. Were you compressing the audio? What codec? VBR?

[poisondeathray]

If you are using virtualdub, the only container possible is AVI, which doesn't fully support B-frames. Sometimes this can manifest in studdery playback and lag issues.

The "VD Hack" is supposed to fix B-frame lag, but I think it's far from perfect. Perhaps one option would be using virtualdubmod and put it into a .mkv container?

Personally, I would use x264 CLI encoder and put it into .mp4 or .mkv, but you are stuck with the VFW version (in AVI) if you are using virtualdub

[jagabo]

The VFW version also encodes interlaced chroma channels incorrectly.

[boblin2]

I found that using a Target Bitrate of 3,200Kbit/s produced a file of a similar size to using CRF = 18.

Maybe for the file you tested. But CRF will always deliver the same quality (relative to the source) regardless of frame rate, frame size, and nature of the video. Multipass VBR will not. With CRF and CQP you specify the quality and the file turns out to whatever size is necessary to give that quality. In multipass VBR you specify the file size and the video turns out whatever quality is necessary to achieve that size.

OK, I'll do some tests and see which gives me the results I want. The one-pass approach is a significant advantage in itself.

If 'Max consecutive' is not equal to 0 AND 'Threads' is not equal to 1, THEN 'VD hack' has to be checked ELSE the video will lag the audio. That was the case on my machine anyway.

How much sync error were you seeing? I've never noticed this. Were you compressing the audio? What codec? VBR?

The delay is approximately 150ms on a PAL video. The audio was uncompressed, and I even tried it compressed to isolate te problem. Hovering the mouse pointer over the VD hack check box brings up this message: "VirtualDub hack for fixing B-frames lag (also multithreading lag). CAUTION! Use only with VirtualDub...". The Codec was whatever the x264vfw H264 is. I used 2 pass encoding with the bitrate set at 4,500Kbit/s.

In an earlier post you recommend that I turn off B frames. How is this done using the application that I'm using? Having said that, everything else I read suggests that B frames are good, but are they only referring to good from a compression point of view or good in terms of quality?

[boblin2]

The VFW version also encodes interlaced chroma channels incorrectly.

How can you tell this, and what problems will it cause me later, if any? Thanks.

[PuzZLeR]

Boblin2, you're probably feeling overwhelmed right now, but I will make a couple of points.

VFW is outdated. And as well, H.264 in an AVI is an abomination. The aging AVI container wasn't built to "see" that far with the modern codec's ability to reference multiple frames or add more b-frame options, etc. (much like poison touched on). You can still play the video but you're stifling it considerably.

As well, the CRF function will indeed give you varying file sizes, some big, some small. Every movie's needs is unique depending on its motion scenes, flashing, rate of change, etc. You will never fit the same clothing size on every human being, and the same applies to video too. Don't fall into the trap of being an overweight individual trying to fit into a smaller clothing size because he/she is in denial. :P

When you're deciding a target bitrate of 3200kbps it's only a guess, not something that is natural for the video itself. You may be depriving it or bloating it. The CRF function will give you your quality at the lowest possible bitrate without wasting any data. But maybe CRF=18 might be a little high, I admit, and most people are fine with 20 or 21 to get a nice file size.

However CRF=18 is "near lossless" encoding when a video clip is quite special.

[jagabo]

The VFW version also encodes interlaced chroma channels incorrectly.

How can you tell this, and what problems will it cause me later, if any? Thanks.

The problem is the flip side of the chroma upsampling bug that was so prevalent in DVD players in the past:

http://www.hometheaterhifi.com/volume_8_2/dvd-benchmark-special-report-chroma-bug-4-2001.html

The error is occuring while downsampling for compression, not upsampling after decompression. This results in colors from the two fields blending together.

In YV12 encoded video the chroma channels are stored at half the resolution of the luma channel. So a 720x480 frame of video has the luma channel at 720x480 and the chroma channels at 360x240. Typically an encoder will convert RGB (4:4:4) to YUV (4:4:4) then downsize the U and V channels (4:2:0). If the scanlines aren't separated before downsizing the colors from the two fields get blended together.

Here is a correct example:



On the left is a crop from an interlaced frame of video. In the middle is one field (converted to a full height frame by duplicating scanlines), on the right is the other field. On an interlaced TV you would see these two fields one after the other. First the red box over a white box, then a blue box over a black area.

But with x264vfw's chroma downsampling bug you get this:



The colors of the two fields have been blended together. When you see this on an interlaced TV you will see a purple box over a white box, then a darker purple box over a black area.

Of course, I chose these two fields to show the problem at its worst. With most real world video it is not nearly as obvious.

Another thing to keep in mind: VirtualDub does not upsample interlaced YV12 video correctly for display and RGB filtering. It has the same upsamping bug that many DVD players had. It treats all YV12 as progressive.

[boblin2]

jagabo, thank you for such a detailed explanation.

Putting aside the color space issue for now, I chose to press on with x264vfw so that I would have something to compare the other encoders against.

The following four images show how, using x264vfw and VirtualDub, I was able to achieve my goal of a video file that was both considerably smaller than the original DV file, while at the same time having considerably better image quality than an MPEG-2 file:





Notes:
- Values used for 'Ratefactor' and 'Max IDR' were selected to increase image quality and future editability, respectively.

- In VirtualDub, if 'Threads' is not equal to 1, then 'VD hack' must be checked; else the video will lag the audio.

- SAR (Sample/Pixel Aspect Ratio) = 16:15 for 4:3 PAL, and 9:10 for 4:3 NTSC.
ProCoder indicates PAL DV has a PAR of 16:15 (1.067), and Vegas indicates PAL DV PAR is 1.0926 (12:11). Putting aside this difference, and using 16:15 for PAL worked fine. Using 12:11 resulted in the video being scaled incorrectly in the ATI Media Player.

- B-frames are set at zero because although they improve compression, in an AVI file they cause problems for editors, including Sony Vegas.

[jagabo]

- B-frames are set at zero because although they improve compression, in an AVI file they cause problems for editors, including Sony Vegas.

The major reason B frames improve compression is because they are compressed with higher quantizers (less quality). The idea being that you can sacrifice a little quality for short periods of time (the quality is cleaned up again by a later I or P frame) and you won't notice it much.

In theory, B frames can compress better because they can reference frames before or after them. But in my experience, if you encode B frames with the same quantizer as I and P frames, you get very little additional compression.

Another reason to avoid B frames for video which will later be edited and recompressed is that the drop in quality, and later restoration of quality, is seen by the later compression codec as more changes in the picture, requiring more bitrate to encode!

- In VirtualDub, if 'Threads' is not equal to 1, then 'VD hack' must be checked; else the video will lag the audio.

I was finally able to reproduce this. Someone once uploaded a test video that was all black except for a single test pattern frames and accompanying tone bursts, once every second. After encoding with x264vfw it was clear that the video and audio were out of sync. But only with some players.

Some background:

The old Video For Windows (VFW) API is based on a one-frame-in-one-frame-out model. VFW reads in one compressed frame, decodes it, then turns the decoded frame over to the the calling program. This causes a problem for B frames which cannot be reconstructed until the a later I or P frame has been decompressed.

Consider the common four frame sequence of IBBP. The first frame is an I frame, a self contained image much like a JPG image. The fourth frame is a P frame. It contains only the differences between itself and the preceeding I frame. The two B frames may reference both the preceeding I frame and the later P frame. For example a B frame may say "this part of the image is the same as the preceeding I frame", and "this part of the image is the same as that later P frame." So the B frames can't be reconstructed until both the I and P frames are decoded.

The workaround for this is encapsulate the two B frames with the P frame and add two null frames to make up for the missing frames: I(BBP)NN. The BBP sequence is stored as if it was one frame in the AVI file. The decoder keeps track of this and stores three intermediate frames internally, returning them one at a time as requested by the calling program.

Now when VFW reads the first frame the decoder decompresses the data the I frame returns it to the calling program. When VFW reads in the second frame the decoder gets the compressed data for all three of the next frames. It decodes all three and stores them internally, and returns only the second frame, the first B frame, to the caller. Then VFW reads the first null frame. The decoder ignores this and returns the second B frame. Finally, VFW reads in the second Null frame and the decoder returns the P frame. The process repeats for every BBP sequence.

So it's not really that AVI is "an abonimation". AVI, as a storage format, can handle I, B, and P frames. The problem is in VFW's handling of AVI files and the extent to which programs have to go to to get around it.

Without this workaround the x264 decoder probably delays frames long enough to get around the B frame issue. So when decoding starts it gets the compressed data for the I frame stores it internally, and returns a black dummy frame to the caller. It does the same for the two B frames. Finally, when it gets the P frame data it decodes all four frames (IBBP) and returns the picture for the I frame. As subsequent frames are read and decoded the output frames continue to lag behind. Some players know to delay the audio as well, some don't.

This doesn't explain why the number of threads used during encoding would make a difference. I suspect that the multithreading code broke something in the encoding that alerts players that the audio needs to be delayed.