Quantization Matrix of TMPGEnc

[sanjayk]

I have been using TMPGEnc Plus for quite some time for converting various formats to MPEG1 & MPEG2. But everytime I have used the "default" option in the quantization matrix settings. There is one option called "MPEG Standard". I tried converting some files using this option but IMHO, I did not observe any significant difference. But the time taken to encode was slightly more.

Is there really any benefit using "MPEG" option in the QM settings ? Has anybody tried this before ?

[sanjayk]

Well, I guess nobody has tried this before

[Shadowmistress]

I've tested it a while back when I was messing with the quant table numbers. Mpeg standard just changes the right side of the tables to all 16. This side of the table controls how much compression to apply to moving objects through I,B,P frames. (The left side controls compression of light and color.)

If I'm not mistaken, mpeg standard would make your filesize slightly bigger because it's using less compression for non moving areas of the screen. On the other hand, if you're watching animation where sudden screen changes occur all over the place and no area should really be classified as non moving, there would be less "noise" occurring randomly by using this setting.

Use for high motion stuff to retain quality when you're getting noise with the default setting. It will make the file bigger tho.
Use default for all other normal stuff.

And for all you quality freaks out there, if you don't care about filesize, try changing the quant tables to all 8's. Both sides, every box. Makes a big ass file, but the lower the number the better the quality (and the less compression so bigger size).

[lordsmurf]

I found the MPEG STANDARD blocky.

The CG/ANIMATION was odd too.

The DEFAULT seemed fine. I have tried to tweak it, but didn't see much difference.

The KVCD MATRIX is not very good either. Not blocky, but the color palette seems squashed and the darks overpower the lights.

After all this, I choose to use DEFAULT.

Or another encoder, when filters are not needed.

[Shadowmistress]

I don't know all that much about the finer workings of the quant table. All I know has come from reading and much much practice and observation, but I've developed a rudimentary understanding of it. Of all the info out there, I found this place to be a great place to start for explanations to newbs. http://www.iis.ee.ic.ac.uk/~frank/surp99/report/kht97/work.html
After that, I did tests by taking different clips, one with lots of really dark scenes (batman) and one with lots of really light scenes (waterworld) and also using action and still scenes from both. I turned the biterate down really low, (and I mean almost to the point of tmpgenc choking on it) and messed with the numbers just to see a difference. That's where my opinion comes from.

I think CG/Animation sucks because it changes the whole left side of the table to all 32's. Now 32 compared to 8 is too low a number for decent quality, even tho it compresses less than 83.

The default matrix isn't perfect either. I can see why some "K" fans have tried to mess with it. The problem is, ultimately you'd need to keep changing matrixes depending on your source video to achieve "perfect" quality. You'd need a good grasp of how the tables work to see where you're wasting bytes on quality and clipping bytes in compression since movies come with varying degrees of motion and light strength.

[animatron]

The KVCD MATRIX is not very good either. Not blocky, but the color palette seems squashed and the darks overpower the lights.

If your color space is wrong, it's because you're doing something wrong

Have you ever wondered why the MEncoder Group, the people that designed Mencoder and adopted the source repository of FFmpeg incorporated kvcd.net's notch matrix as a standard option in their source tree
Look at the official document:

7.10. Custom inter/intra matrices

With this feature of libavcodec you are able to set custom inter (I-frames/keyframes) and intra (P-frames/predicted frames) matrices. It is supported by many of the codecs: mpeg1video and mpeg2video are reported as working.

A typical usage of this feature is to set the matrices preferred by the KVCD specifications.

The KVCD "Notch" Quantization Matrix:

Intra:

 8  9 12 22 26 27 29 34
 9 10 14 26 27 29 34 37
12 14 18 27 29 34 37 38
22 26 27 31 36 37 38 40
26 27 29 36 39 38 40 48
27 29 34 37 38 40 48 58
29 34 37 38 40 48 58 69
34 37 38 40 48 58 69 79

Inter:

16 18 20 22 24 26 28 30
18 20 22 24 26 28 30 32
20 22 24 26 28 30 32 34
22 24 26 30 32 32 34 36
24 26 28 32 34 34 36 38
26 28 30 32 34 36 38 40
28 30 32 34 36 38 42 42
30 32 34 36 38 40 42 44

Usage:

$ mencoder input.avi -o output.avi -oac copy -ovc lavc -lavcopts inter_matrix=...:intra_matrix=...

$ mencoder input.avi -ovc lavc -lavcopts
vcodec=mpeg2video:intra_matrix=8,9,12,22,26,27,29,34,9,10,14,26,27,29,34,37,
12,14,18,27,29,34,37,38,22,26,27,31,36,37,38,40,26,27,29,36,39,38,40,48,27,
29,34,37,38,40,48,58,29,34,37,38,40,48,58,69,34,37,38,40,48,58,69,79
:inter_matrix=16,18,20,22,24,26,28,30,18,20,22,24,26,28,30,32,20,22,24,26,
28,30,32,34,22,24,26,30,32,32,34,36,24,26,28,32,34,34,36,38,26,28,30,32,34,
36,38,40,28,30,32,34,36,38,42,42,30,32,34,36,38,40,42,44 -oac copy -o svcd.mpg

Link here: http://www.mplayerhq.hu/DOCS/HTML-single/en/MPlayer.html#custommatrices

And professional video developer and Mencoder developer Steven Shultz answers here:

 > In short - TMPGEnc matrices always produce smaller files than default ones, 
 > KVCD - always smaller than TMPGEnc, in my experiments difference in file size 
 > was up to 20%.
 
 	That is the same thing I am seeing.   KVCD"s tables result on average
 	saving about 16% and TMPGEnc"s tables about 10% compared to the
 	default tables.   That"s well worth the effort.      To paraphrase
 	an old joke "Q: how does a file get big?"   "A: one percent at a time"
 	:)
 
 	Cheers,
 	Steven Schultz

Link here: http://sourceforge.net/mailarchive/forum.php?forum_id=3248&max_rows=25&style=nested&viewmonth=200307

I could give you more links, but this gets tired very quickly

Cheers!

[animatron]

The default matrix isn't perfect either. I can see why some "K" fans have tried to mess with it.

And the KVCD matrix isn't perfect either
It just so happens that it's the one that does the best job on dim lit surfaces, a problem which plagues all MPEG-1/2 encodings with standard matrixes.
This was also acknowledged on another thread on the Mencoder's mailing list by the developers.
That's really the key and advantage of the notch matrix.

[Shadowmistress]

@ animatron - we've had this same discussion about 4 months ago. Nobody here is talking about crunching the filesize, we're talking about better quality than mpeg default, and kvcd will simply NOT cut it.

@ lordsmurf - try feeding these numbers into the quant table and see what happens. It will look way better but produce a larger filesize.

Let me know what you think.

[sanjayk]

shadowmistress - thanx for such an elaborate discussion and information on QM. I'll try your QM values an see for myself.

[animatron]

@ animatron - we've had this same discussion about 4 months ago. Nobody here is talking about crunching the filesize, we're talking about better quality than mpeg default, and kvcd will simply NOT cut it.

I'm very sorry to dissapoint you Shadowmistress, and others too!, but kvcd's notch is "transparent" to the human eye, and it DOES cut it
To put a stop to the fallacies that I keep reading on this site about kvcd's matrix, here are a couple of videos to finally clear and put an end to this issue. (links at bottom of post. each is 22 seconds in length)
It you or ANYONE on this site can tell the difference between the two videos, and I mean ANY difference in colors or lost details, I'll be happy to make an appointment with an ophthalmologist first time tomorrow morning.
You can zoom in to your heart's delight with Virtual Dub, or do whatever you want to analyze the files. The only thing that you will probably find is LESS blocks on some dark spots on the kdvd clip, and that's the notch doing it's work.
Go ahead and zoom to 300% in vdub, and put a big magnifying glass on front of your CRT, and then you'll see very slight differences which are just impossible to see on any TV set by the naked eye.
Other than that, the videos are identical, and that's what I call transparent.
There is not a single difference between the two PAL Interlaced encoded files, except that the KDVD video is almost 2 megabytes smaller than the one encoded with the MPEG standard matrix.
Also, there were zero filters applied. The script used was this:

Mpeg2Source("C:\DVDBackups\DD\duran.d2v")
Undot()
Limiter()
BicubicResize(704, 574, 0, 0.6, 8, 0, 704, 576)
AddBorders(0, 1, 0, 1)

That was it!. Nothing more and nothing less.
For reference, both video were encoded with tmpeg at a cq of 85, a maximum bitrate of 8,000kbps and a minimum of 300kbps.

Cheers, and happy encoding!

----------------------------------------------------------------------------------
Encoded with MPEG Standard Matrix.
Duran-Duran-MPEG-Standard-PAL.m2v
http://s24.yousendit.com/d.aspx?id=0UWDWYUI3FX8V3J3ET9820DP8R

Encoded with KVCD's notch Matrix.
Duran-Duran-KDVD-PAL.m2v
http://s51.yousendit.com/d.aspx?id=0YTLYT0CTACZ22P59QO4KGDB6Z
----------------------------------------------------------------------------------
Edit: If anyone can download and mirror the files, I'll be grateful, because I don't have any site where I can upload them and yousendit.com only allows a certain amount of downloads.

[adam]

I don't see much difference between those two clips, but I wouldn't expect to with such a low quality source (I assume) and such close filesizes. I would expect that if you lowered the bitrate and made the "mpeg standard" one 2MB's smaller it'd look the same as the kdvd one. Honestly both of those clips look poor, but it could just be their haircuts.

In my opinion, customized quant matrices make very little difference so long as your source is decent quality and you use a reasonable bitrate. If you absolutely must cram alot of content on a disk or, in the case of the original poster, your source quality is already lacking, than customized quants, including the notch one, are worth checking out.

Try some of the ones in the link at the very bottom of this page. They help in low bitrate encodes, but like I suggested earlier, the best solution is to just raise the bitrate.

http://www-users.mat.uni.torun.pl/~jawor/matrix/matrices.zip

[animatron]

Honestly both of those clips look poor, but it could just be their haircuts.

It may very well be
The source is a VOB from a DVD, and it looks identical to both clips. Probably because it's an old 80's video, so it's not top quality.
Frankly, the lower the bitrate, the better the notch matrix video compared to the standard matrix. At least that's what I get on every low bitrate videos I encode.
We could probably say that the notch "scales down" better than the standard matrixes, probably because it lowers quantization on the low frequency bands.
The standard matrix will try to quantize dark areas harder, which produces the infamous DCT dancing blocks. This is crearly reduced with the notch.

[ronnylov]

Try some of the ones in the link at the very bottom of this page. They help in low bitrate encodes, but like I suggested earlier, the best solution is to just raise the bitrate.

http://www-users.mat.uni.torun.pl/~jawor/matrix/matrices.zip

How do I open the xcm files?

In TMPGEnc the matrices are stored in the ini file so the easiest way to put them into TMPGEnc would be editing the ini file.

Many matrices are optimized for lower bitrates but some are good for higher bitrates. I have done some tests with the mb1 DV matrices and they give a sharper image when encoding from interlaced sources at relative high bitrate (6000 kbit/s average or higher).

By rising the numbers more data are thrown away in the encoding process and vice versa. The human eye is less sensitive to fine details so that is why higher numbers can be used in the lower right corner. You can say the matrices are some kind of filter in the frequency domain. At lower bitrates you need to throw away a lot of data and matrices with high numbers in the lower right corner may make it easier for the encoder but you may get other problems like mosquito noise or blurry picture.

At a high bitrate the encoder might be bitrate saturated and you won't gain any quality by raising the bitrate. By using matrices with lower numbers, expecially in the lower right corner you may allow the higher bitrate to be used to encode finer details of the picture or getting less mosquito noise.

The standard MPEG matrices are a compromise that works fairly good in most cases. In many cases it's not worth modifying the matrices because what you gain in one way you loose in another way and the only way to really increase the quality is to encode at a higher bitrate.

[Shadowmistress]

I agree with adam and ronnylov. The higher the numbers in the lower right corner of the boxes the more quality you retain by throwing out less bytes.

animatron, again, we're not talking about getting the best quality for the smallest filesize, we're talking about getting better quality than mpeg default for any filesize. Something that looks better to the human eye, not something that "fools" it. And you'll just have to face up to the fact that some people's eyes are simply not being fooled.

[ronnylov]

It throws away more bytes when the number is higher. The number is a threshold and if the level of this frequency combination is above the threshold then it's encoded, if not it's thrown away and lost forever. MPEG is lossy. It throws away less important information and tries to keep the information where it is needed.

At a given bitrate it must throw away a given amount of data to keep the bitrate. So if it throws away more at higher frequencies it will keep more at lower frequencies where it is more visible. It is possible to optimize the matrices for a given source video but perhaps you could as well encode at a lower resolution or just use some kind of softening filter to decrease the high frequencies before the encoding.

This document gives a good overview of MPEG encoding:
http://www.snellwilcox.com/knowledgecenter/whitepapers/papers/mpeg_video.pdf

Look at page 5, 6 and 7 and you can see how the quantize matrice is used to match the sensitivity of noise in the human eye. On some type of video (like animation) certain types of noise (or encoding artifacts) are more visible and it should be possible to make matrices that is better than the default by optimizing but it is probably quite difficult to find a good compromise for all types of video.

[adam]

ronnylov, I meant to link to the page not the file.

http://www-users.mat.uni.torun.pl/~jawor/matrix/

You would use the custom quantization editor to open the files then just plug those values into your encoder.

[Shadowmistress]

Daaaaamn, that second scan order... is that the how it reads inter-frames?

That's messed up.


The numbers look about right on the 2 cd matrix to fill up the second disk. My example above would just spill over into a second, but not fill it.

[animatron]

Daaaaamn, that second scan order... is that the how it reads inter-frames?

That's messed up.

Scan order is not related to inter on intra matrix.
It doesn't matter what scan order is used. It's the matrix weighting (frequency domain) of each matrix position that affects the discrete cosine transform.

[Shadowmistress]

Scan order is not related to inter on intra matrix.

Dude, that doesn't even make sense. They're two different things. What makes you think I was talking about an intra matrix? Do you even know what I was referring to?

It doesn't matter what scan order is used. It's the matrix weighting (frequency domain) of each matrix position that affects the discrete cosine transform.

Ok, what site did you copy and paste that from? :P

[lordsmurf]

@SM
I'll definitely plug your numbers in the near future. Thanks.

@animatron
Sorry, you're never going to win this one. Kwag's stuff looks like shit compared to better methods. I'm not going to sugar coat it to sound "nice" or "walk eggshells to make everybody happy". He made his stuff years and years ago, for the sole purpose of copying commercial DVDs to CDs. The days of cramming video are over. If his methods were so super-uper-duper, they would be implored by studios. They are not, because they suck. Let's look at them for what they are, a way to not totally deteriorate video with squishing it (but remember that a good bit of quality deterioration is still going on here).

@adam.
I think we agree for once. Matrices make little difference, as compared to source quality, bitrate allocation, and the encoder used.

[animatron]

Ok, what site did you copy and paste that from? :P

It's common engineering knowledge
But I'll give you a link so you can bookmark it, which explains the DCT process: http://bmrc.berkeley.edu/research/mpeg/faq/mpeg2-v38/faq_v38.html

@lordsmurf,

Stop trolling! ( http://www.urban75.com/Mag/troll.html ) and post constructive feedback
I'm sure if I do a search on this site for "kvcd" or "kwag" or anything related +"lordsmurf", your posts will be #1 !!!!!

[Shadowmistress]

I've read parts of that website before, but had forgotten about the freaky alternate scanning pattern used for interlaced sources, not for inter matrix blocks as I asked above.

And animatron, you're the pot calling the kettle black when you bring up trolling. All it took was me mentioning "K" fans to get your panties in a bunch. :P

Chill out already.

[animatron]

All it took was me mentioning "K" fans to get your panties in a bunch. :P

Hehe, look at the top of this thread (fourth post), and you'll see who really started the "K" issue

Cheers!