XviD Quantization type, H.263 vs MPEG; and GMC for hand recorded material

[TCmullet]

...I had a player several years ago that choked on any XviDs that were over 4000 for about a second. It would stutter and freeze temporarily. The loader (the DVD-ROM part of it) gave out and I replaced it with a retail Lite-On and another test I did that maxed out at over 7000 played flawlessly. I have no idea of the max speed of the 2 although I suspect the Lite-On was faster (2x .vs 4x maybe?), which could serve to confirm what jagabo said - that the spin rate of the loader is important.

THIS makes sense. Therefore I have to use a disc.

In your test, isn't the XviD encode near to the size of the source? If so, wouldn't it make more sense to keep the source unchanged?

Not sure what you mean. I never mentioned the size of the source or what form it's in. The *direct* source is in HufYUV, 24fps, 18GB. The real original (at my end) was an mpeg2 file captured from cable at 10Mbps max bitrate VBR, 30fps. But if the signal was really good, a lower mpeg2 rate could have sufficed. Yes, some movies do capture to less than 4GB, so I could do what I've done for years w/TV shows; burn the mpeg2 program streams to disc. But now that I have a player that does Xvid from data discs pretty well (and some extra hardware to let Vdub run for hours at a time), I want to improve the picture. IVTC, logo removal (where needed), letterbox cropping, and perhaps a bit of noise reduction if needed; these are most of the reasons I want to go to Xvid. (I've been wanting to do it since '03, but only now find it feasible.) Of course one big reason would be cases where the mpeg2 would be bigger than 4.37GB. I want to get each movie down to 1 disc, and Xvid allows me to do that better than mpeg2 captures.

...Or switch to archiving using x264 which compresses much better?

I'm open to that, but never heard of it. Are you saying x264 is a quantization matrix? Or is it a totally different codec? If so, then that's not my goal (unless someone changes my mind). While archival (as I referred to it earlier) is a separate goal from movies, I *have* preferred to do both movies and archival in a format that can be viewed by a DVD player on a TV. If to get the best quality, I have to sacrifice DVD player-ability, I would still prefer to use the same codec (Xvid) as the movies. I hope this makes sense.

Also, my recommendation when using those high-bitrate matrices wasn't to use Q1, but 3-5, depending on the matrix used.

Yes, I know you weren't recommending Q1. But I still wanted to see what it would do. I get the impression it's better quality than Q1 with the original two.

In the case of the Sharktooth's EQM v3UHR, I daresay you could use Q5 and not be able to tell the difference from the source. But I could easily be wrong, not having tested myself. Again, though, you shouldn't ever have to use Q1, especially not when using a high-bitrate custom matrix. You asked so I gave you my opinion.

You may be right about Q5. But I can't test that right now, as I can't get a new monitor. My gut tells me there *must* be a difference, or the bitrate graph wouldn't be so jacked up w/Q1. But for now, I'm only working on movies, so it may be that the disc buffering and disc speed in the player will force me to your higher Q levels. Will try 1 or more discs.

[jagabo]

x264 is an MPEG 4 Part 10 encoder. AKA: AVC, h.264. DVD players generally do not support this. Many media players like the Western Digital WDTV series do. And some Blu-ray player will play un-authored discs with h.264 M2TS files. You're encoding times will increase quite a bit unless you're using a quad core or better CPU.

[manono]

I never mentioned the size of the source or what form it's in.

Ah, but you did:

I do high bitrate mpeg2 captures of video.

You mentioned the source and using the high bitrate matrices at Q1. I was making an educated guess that the XviDs were approaching the size of the sources. I don't quite understand why you make an intermediate HuffYUV AVI though, when you can directly reencode from the source MPEGs via the DGDecode.dll, MPEG2Source, and an AviSynth script.

[ronnylov]

There is never any need to go lower than 2 with the quantizer of xvid. I have read discussions somewhere taht Q=1 is only waste of space.

Another thing to mention is that you should do some testing to find the limits of your player (I see that you already did some tests). When you find the limits you can use vbv and profile settings to make sure that your encoding stays within your chosen limits. But when I was doing this on my player the xvid builds were broken regarding staying within vbv limits so I had to use a special build and a special method to do my constant quality encodings using 2-pass encode with the bitrate on second pass the same as the first constant quality pass. But this is only necessary if you actually get playback problems.

Read more here:
http://forum.doom9.org/showthread.php?t=124673

I do not know if this issue has been fixed in xvid or if we still need to use the modification made by plugh to get this working... I have now moved on to x264 and replaced my old xvid-player with a htpc.

EDIT: The modification of xvid encoder mentioned above is the "altpass2" option invented by plugh.
http://forum.doom9.org/showthread.php?t=118419

[jagabo]

If you're using b frames and a target quantizer of 2 (with the default settings) the b frames will be encoded with a quantizer of 4 . If you use a target quantizer of 1 the b frames well be encoded with a target quantizer of 2. So the b frames will look quite a lot better.

[TCmullet]

I've done test. And I've created discs!

Here's my data, some previous reported, but augmented.

Movie: 20 Million Miles to Earth, 720x344

Q=1.0 Quantization type=H.263 resulted in 3.23GB
Q=1.0 Quantization type=MPEG resulted in 2.80GB
(add greyscale filter, here and from now on w/this movie)
Q=1.0 Quantization type=MPEG resulted in 2.75GB Max bitrate=17231 Avg=4443 kbps

Switch to custom quantization matrix "Sharktooth's EQM v3UHR.cqm"

Q=1.0 5.85GB Max bitrate=28625 Avg=9831 kbps (Whoa!!)
Q=1.5 4.11GB Max bitrate=24415 Avg=6809 (I must stick to 4.00GB and smaller.)
Q=1.6 3.88GB Max bitrate=22613 Avg=6413
Q=2.0 2.86GB Max bitrate=16807 Avg=4647

The highest bitrate peaks: Q=2.0
00:02:41 (16807)
00:56:33 (16079)
01:05:09 (14241)
01:07:32 (13494)
01:13:28 (11600)

This Q=2.0 file played fine at each of the 5 high-points, as somewhat expected. Obviously, the peaks are *transient* in nature. Otherwise a player is not required to tolerate a sustained rate over 9.8M. (Forgive me if I can't remember whether that include audio or not.) I'll have to carefully check the file when I get around to doing the Dances with Wolves stampede scene. (I got that movie mainly because of that awesome segment.)

The highest bitrate peaks: Q=1.6
00:02:40 (21503)
00:56:33 (22613)
01:05:09 (19682)
01:07:32 (16310)
01:12:56 (13571)

With Q=1.6, the player played most points on the disk properly, but the one at 2:40 stuttered. It was a brief scene of zooming forward over water, not exactly "transient". (The spaceship was about to crash into the sea.) This may explain why the higher point at 0:56 didn't hiccup. It must have been more transient than 2:40.

This test makes it clear that I must stick with Q=2.0 or higher. My test shows that when the bitrate peaks at over 20M, it's too fast for the disc to keep up.)

Thanks to all of you for the various valuable insights, especially for pushing me outside my comfort zone of "standard resources" to try a custom q-matrix.

My thoughts about the vbv stuff (which I haven't been able to get clear, or even find in the Xvid switches) which was meant to prevent unacceptable bitrate spikes, will be alleviated by me by checking the bitrate profile noting the 2 or 3 highest peaks (I did 5 for my 2 tests above), then play the burned disc at those spots. If I ever find a hiccup, it is then that I will reencode to 2.1 or 2.2, bumping it up as needed. Of course bumping it up may be needed for other reasons, such as a film being very long and full-screen. But heck, it may be that in some cases of 2 short related films, I'll be able to burn them to one DVD-R disc! (But my general goal is to have each film on a separate disc for easier management.

(Will respond to some of ya'lls comments separately.)

[TCmullet]

x264 is an MPEG 4 Part 10 encoder. AKA: AVC, h.264. DVD players generally do not support this. Many media players like the Western Digital WDTV series do. And some Blu-ray player will play un-authored discs with h.264 M2TS files. You're encoding times will increase quite a bit unless you're using a quad core or better CPU.

I knew nothing about this, so thank you, jag. I found the Vdub codex, but it's so full of a zillion switches, I can't tackle that now. (There's probably a good tutorial somewhere.) However, the real reason for now will be to keep files 'Philips-compatible'. But when I get to a point where I want better quality for serious preservation only (no player needed), I'll look into it. The concepts of smaller macro-blocks intuitively sounds neat, even though I know frightfully little about all that high math stuff. (In college, I switched from math to comp-sci.)

[TCmullet]

I never mentioned the size of the source or what form it's in.

Ah, but you did:

I do high bitrate mpeg2 captures of video.

But that was ages ago, and I forgot what I said this morning!

[jagabo]

I think many of the general rules of Xvid encoding (eg, the MPEG matrix is sharper than h.264 and generates larger files in Target Quantizer mode) fall apart when you start working at extremes like Q=1.

[TCmullet]

I don't quite understand why you make an intermediate HuffYUV AVI though, when you can directly reencode from the source MPEGs via the DGDecode.dll, MPEG2Source, and an AviSynth script.

Well, to answer that, I'll have to give some background. (Hope you find it interesting.)

I've only played with Vdub occasionally, that is, get intense about it for a few days or weeks, then drop it for a few to MANY months. From '01 to '07, I pretty much dropped it each time I started playing with it, as I would always come to the conclusion that deviating from straight mpeg capture wasn't a profitable or feasible endeavor. However, I always thought the Vdub concept was neat. When anyone would mention AviSynth, I might freeze up worse than a power surge in a DVR. You see, I really feel good seeing those 2 panes there, the before and the after. With Avisynth you get none of that. All the dynamicness gone.

Believe it or not, I used to hate Windows, before Win95. But in '97 I got hooked on GUIs. So Avisynth seems the pits.

There's a more compelling reason to use Vdub. Delogo appears to be the only reasonably "easy" and "effective" way to remove those PESKY (understatement) logos. Removing them is a big reason for re-encoding from the Mpeg capture in the first place. You HAVE to do that in Vdub. And experience (and advice) tells me that it has to be the first processing done.

Now you might say, "put all the rest of your filters in place and create the Xvid AVI all in one pass". Not so fast. The very next thing to do is IVTC. I interacted with "Phareon" signficantly over usage and features. He now has it to where it's "usable" in my opinion. I've never done a test, but am leary about putting it after Delogo in the same run, what with it messing with frame counts/numbers, etc. There may be no problem, I realize.

But in some cases, the telecined material is screwed up enough that Vdub's IVTC can't be done. Months ago, I was advised to use the Telecide/Decimate filters instead. Once you understand how they work, it's really helpful for cases of material edited after it was telecined. Or even just a dropped frame will screw up the Vdub IVTC from that point forward.

So you can see now that at least in this case, an intermediate HUF file is needed. (Vdub and Delogo to Huf, then Huf via A. script back into Vdub). In the cases where there's no logo and the IVTC filter now in Vdub works flawlessly (due to a flawless film transfer/capture), yes I can make one Vdub run. BUT, even then, there is reason to utilize the lossless HUF file as an intermediate stage. All my research reported here in this thread is a perfect example, even though I don't need to do the research any more. All the painstaking delogo work (using 3 sets of Delogo frames) is behind me; the IVTC is behind me (regardless of whether from Vdub or Telecide). And any general video filtering is behind me. All three of those items collectively take much CPU time and need to be done just once. Once they're all done and I have a clean 24fps HUF AVI, I can quickly (relatively) run as many Xvid encodes as I need, to create the one I need WITHOUT repeating all the prior processing.

Btw, 2 years ago, I found an important use for Avisynth, which I'll be using a lot of in the future. My dad shot TONS of 8mm film. I'm in charge of getting it converted, and we have a small outfit in Maine doing a great job doing single frame capture straight to DV files. We have him do NO pulldown (which he normally does for most people), so that I can have the option of doing filtering w/Vdub or some other AVI editor. Where Avisynth comes in is that I have to get it from 16fps to 30 at some point before mpeg encoding and DVD authoring. Somewhere (probably on the Vdub forum) some kind soul pointed me toward a simple A. script that will do it. (I think his was 18fps.) I studied it carefully and invented the script to do 16-to-30. I've already done one batch of films and it's great to know I have 16fps progressive footage and can telecine to 30 (29.97) and make the DVDs. I can add further features and filters to the original in the future and redo the telecine process using the tiny script again as needed. So I do see some value in Avisynth, but I just can't get excited about attempting nearly everything in it, when Vdub is so much easier to use.

I think I've addressed everything related to your question. Let me know any further questions or thoughts. I'm now going to delete all my "20 million" (pun intended) work files and tackle "The Time Machine". Vdub will be real nice, as the right audio channel was too soft, so I'll strip the .wav off, fix it in CoolEdit, and put it back with no detriment to the video or audio. Neat.

[TCmullet]

I think many of the general rules of Xvid encoding (eg, the MPEG matrix is sharper than h.264 and generates larger files in Target Quantizer mode) fall apart when you start working at extremes like Q=1.

Thanks, Jag (oops it's "jag" not "Jag", even though it ought to "Jag" IMO). THAT was the one last (and tiny) issue that was still puzzling me, even tho less relevant.

Funny you call Q=1 an extreme. During one of my earlier journeys with Vdub (Divx experimenting in '03), I asked someone on their forum "what about hi-rate encoding to archive important footage?". I was told to set Q to 1 (meaning don't drop any info). I never followed through on it, after quickly learning that it wasn't any good for my then current task of capturing Enterprise (S.T.) at medium Mpeg bitrates and encode to something smaller (2-pass). Q=1 obviously greatly bloated it when I was trying for smaller space not bigger. But I never forgot the idea, and now I've attempted it using Xvid. (Btw, I still have my mpeg captures of Enterprise, waiting for the day I can resolve obstacles and process them. If I ever do that is.)

Because Q is an indication/controller of how much info is dropped, I would think one would naturally start with Q=1 and work your way away from it, checking space vs. quality loss, so I don't see it as an extreme. But as I know most Xvid/Divx users are attempting to get as many hours of video on a mini-CD as possible, to them Q=1 would seem extreme, seeing at they would have found out VERY quickly that Q needs to be a high value for them. As for me, I've encoded 7 movies so far using the built-in Hxxx matrix. In many cases I was able to use Q=1. In a few, I raised it somewhere to get the file down to 4.00GB. Logically, Q=1 makes sense, but practically I see now it must be avoided (assuming I will use a good custom Q-matrix, which I will). I'm not ready to jump to the 3 to 5 range yet, but will experiment when the need arises.

[TCmullet]

I have some interesting numbers to report. I did repeated encodings of The Time Machine with the custom matrix and a series of Q values. Here is some of it:

Q=2.0 3.60GB (3,778,630KB)
Q=2.5 2.99GB
Q=2.6 2.89GB
Q=3.0 2.25GB

All these looked great. They included a less than good logo removal by Delogo. (The logo is animated, and I had installed 3 different analyse masks and filter instances to roughly break down the animation into 3 phases.) I came to the conclusion after getting such good results with these Q levels and small sizes that I might like to save a copy of the movie with logos intact, in case I would some day find a better way to remove them. So I did more encodings leaving the logos intact, and had some interesting file sizes.

Q=2.0 3.60GB (3,775,837KB - slightly smaller than with logo poorly "removed")
Q=4.0 1.64GB
Q=5.0 1.31GB
Q=6.0 1.07GB
Q=7.0 0.959GB
Q=8.0 0.853GB
Q=9.0 0.775GB

Upon reaching 9.0, I could now save the Q=2 de-logoed version and a Q=9 logoed version on 1 DVD-R. I browsed the quality of all these, and I was amazed that even Q=9 looked pretty darned good. Admittedly I didn't watch anything on a TV (the most stringent test, more strigent than any computer monitor I have). And the source (TCM on digital cable) is very clean. But still, I never expected anything at Q=9 to be even watchable.

So I could use a Q=2 de-logoed and Q=9 non-de-logoed for a DVD-R. Perhaps I could redo the de-logoed at a bit higher than 2 and use the other one something less than 9. But will not bother as you will see shortly...

Now for the reason why I took so long to post this, even though my testing was done by 1/25. I got into my head the crazy idea that I might be able to build on some recent knowledge of C++ that I gained by taking a course in it, and tear into the Delogo source code (and Vdub to an extent if needed). I wanted to see what would be required to modify it to where I could build a system for eliminating ANIMATED LOGOS!!! So all these weeks I've been consumed at every spare moment with learning "real" C++ and modifying Delogo 1.3.2 to remove my animated logos from these movies. I've also been recording too many new movies and have filled up my storage spaces. So now that I have potential hope for removing the logos from The Time Machine better than in January, I must delete all these files from my tests (keeping the original mpeg) so as to promptly process the movies where the letterbox is narrow enough that the logo doesn't invade. (The logo will go away merely with cropping.) My new Delogo version isn't ready yet, but while numerous heavily letterboxed movies are encoding, I believe I'll get it finished. My first real test will be on The Time Machine. It's letterboxed, but not enough to avoid being hit by the pesky logo.