In a very general sense, I understand matrices as a tool to produce more compressible or visually appealing product, depending on many factors, such as bitrate and source material. I also understand that there is no "best" matrix for all situations.
Possibly relevant:
- Are matrices codec-specific? I.e., can I use XviD matrices for MPEG2—or if it is possible, would it be unwise?
- Is there any list of MPEG2 matrices (in plaintext), ideally which also explains which matrix is intended for which purpose?
- Could using a custom matrix break DVD compliance?
- What is a good test methodology for doing A/B comparison between matrices?
- Can a well-selected matrix make a very significant impact on quality, or is the improvement potential necessarily slight?
- Might you recommend a few matrices for me to try, with a brief explanation of where they excel?
- Might you recommend a high-res, relatively uncompressed sample for me to do A/B testing on?
+ Reply to Thread
Results 1 to 26 of 26
-
-
1. Yes, you can use XviD Matrices, but most are of such poor quality (designed to look passable at very low bitrates) that I wouldn't use them for MPEG-2 encoding.
2. Not that I'm aware.
3. If you don't know the basic rules, maybe, not sure.
4. That's very hard to answer. Maybe the subtract filter for AviSynth. It can show the differences between a source and a reencode with a different matrix. Better, I think, is to find out the resulting Q-Factor. If it's within reason when using the best matrix (meaning the least low frequency data thrown away), then you're good to go. I compiled a bunch collected from various retail DVDs and choose which to use based on the Q-Factor test mentioned above (best quality matrix for a given Q-Factor).
5. The differences can be insignificant. However, the basic matrix used by most of the encoders we're likely to use, what's sometimes called the Standard Matrix, isn't very good. It's one you'll never see in a decent Hollywood movie on DVD. Most of the matrices included in the free HCEnc are garbage XviD ones, but it does have a nice selection of decent ones.
6. I'll attach the ones I use at the bottom. You have to remember that the ones in retail DVD movies are for DVD9s, mostly, where I make them for DVD5s (meaning significantly smaller sizes). I use #9 the most. The quality is in descending order, top to bottom. The percentages given are the sizes relative to #4 (sometimes called Fox1) when doing constant quality encodes. Those percentages are accurate only for the specific material using specific settings within CCE. But they'll also serve as a rough guide.
7. Not I.Last edited by manono; 1st Mar 2014 at 19:14.
-
I interleave frames of videos when I want to compare different encoding or filtering. Then I swap back and forth between frames in an editor like VirtualDub.
Code:v0=WhateverSource("original.ext").Subtitle("original") v1=WhateverSource("encoding1.ext").Subtitle("encoding1") v2=WhateverSource("encoding2.ext").Subtitle("encoding2") Interleave(v0, v1, v2, v0)
-
Actually ffmpeg does support custom quantization matrices. However its MPEG-1/2/4 compressors clearly suck when compared to TMPGEnc, HCenc and Xvid, for example.
Might you recommend a high-res, relatively uncompressed sample for me to do A/B testing on?Last edited by El Heggunte; 1st Mar 2014 at 20:42.
-
I also encode MPEG2 for DVD5, so I decided to do some tinkering with #9. Attached are my results.
I honestly can't see any discernible visual difference. The only difference I saw was at 8000k, where using that qmat resulted in slightly smaller filesize.
Maybe you can help me understand this better?
Here's my ffmpeg script for reference:
Code:#!/bin/bash SOURCE=$1 DESTINATION=$2 VB=$3 VF="scale=720:480,\ hqdn3d=2:1:2,\ pp=ac" # #9 50% INTRA_MATRIX="\ 08,10,10,12,13,15,16,20,\ 10,10,12,15,17,19,20,20,\ 10,12,15,17,19,20,20,23,\ 12,15,17,19,20,20,25,25,\ 13,17,19,20,23,25,25,27,\ 15,19,20,23,25,27,27,30,\ 16,20,20,25,25,27,30,35,\ 20,20,23,25,27,30,35,45" INTER_MATRIX="\ 12,14,17,18,19,20,24,28,\ 14,16,17,18,19,23,27,32,\ 17,17,18,19,20,27,30,37,\ 18,18,19,20,27,30,35,37,\ 19,19,20,27,30,35,37,40,\ 20,23,27,30,35,37,40,44,\ 24,27,30,35,37,40,40,44,\ 28,32,37,37,40,44,44,48" # Pass 1 ffmpeg \ -i $SOURCE \ -framerate 24 \ -vf "$VF" \ -aspect 16:9 \ -intra_matrix $INTRA_MATRIX \ -inter_matrix $INTER_MATRIX \ -dc 10 \ -g 12 \ -b_strategy 2 \ -brd_scale 2 \ -bf 2 \ -lmin 0 \ -lmax '320*QP2LAMBDA' \ -mblmin 1 \ -qmin 1 \ -qmax 31 \ -maxrate 9800k \ -b:v $VB \ -bf 2 \ -pre_dia_size 5 \ -dia_size 5 \ -qcomp 0.7 \ -qblur 0 \ -preme 1 \ -me_method dia \ -sc_threshold 0 \ -sc_factor 4 \ -bidir_refine 4 \ -profile:v 4 \ -mbd rd \ -mbcmp satd \ -precmp satd \ -cmp satd \ -subcmp satd \ -skipcmp satd \ -pass 1 \ -passlogfile ffmpeg \ -threads 4 \ -an \ -f mpeg2video \ -y \ /dev/null # Pass 2 ffmpeg \ -i $SOURCE \ -vf "$VF" \ -framerate 24 \ -aspect 16:9 \ -inter_matrix $INTER_MATRIX \ -intra_matrix $INTRA_MATRIX \ -dc 10 \ -g 12 \ -brd_scale 2 \ -bf 2 \ -lmin 0 \ -lmax '320*QP2LAMBDA' \ -mblmin 1 \ -qmin 1 \ -qmax 31 \ -maxrate 9800k \ -b:v $VB \ -bf 2 \ -pre_dia_size 5 \ -dia_size 5 \ -qcomp 0.7 \ -qblur 0 \ -preme 1 \ -me_method dia \ -sc_threshold 0 \ -sc_factor 4 \ -bidir_refine 4 \ -profile:v 4 \ -mbd rd \ -mbcmp satd \ -precmp satd \ -cmp satd \ -subcmp satd \ -skipcmp satd \ -pass 2 \ -passlogfile ffmpeg \ -threads 4 \ -an \ -y \ -f mpeg2video \ $DESTINATION
Last edited by diprotic; 2nd Mar 2014 at 19:44. Reason: Add ffmpeg script
-
What are you saying? You did them all (or several, to test) at the same bitrate? Then the resulting filesizes should all be similar. Check the resulting average Q. They'll be different. I use the old Tecoltd Bitrate Viewer. Its absolute figures aren't correct, but the relative numbers should be. When the average gets high, artifacts such as mosquito noise, smearing and blocking will occur. It shouldn't be hard at all to tell the difference between the top and bottom ones. How many movies will fit on a DVD5 with an average bitrate of 8000? Try 4000 VBR.
I have to apologize. I looked at your matrix and realized I gave you the wrong one as the one I use the most. It's number seven. That number nine one is for cases when the movie doesn't compress well at all. And the bottom one for cases when it doesn't compress worth a damn.Last edited by manono; 2nd Mar 2014 at 20:14.
-
I tried #4 (what I believe you referred to as Fox1) and #9 (what you initially referred to as the one you use the most). The attached samples are the built-in lavc qmat vs. #9 (includes qmat9 in filename).
Maybe I need to go back to school. Please correct any/all of the below, as you are willing and able:
We compress video because uncompressed is too expensive disk-space-wise. Compression is basically a really complex exercise in determining which bits of visual information aren't as important to faithfully reproduce the original. The best compression will take a budget (bitrate) and "spend" those bits as wisely as possible, with the overall goal of preserving as much original detail as possible while throwing away ("compressing") the not-so-important parts.
Many factors tell an encoder how to spend those bits. One of them is the qmat. Qmats are, in essence, "bitrate budgets," giving the encoder a guide for how to spend bits on high- and low-frequency details. Like all budgets, there is no one-size-fits-all solution. Each qmat provides more efficient compression for different types of sources and destinations.
Q-factor is a numeric representation of how much detail ("high-frequency data") is thrown away. Lower Q means less detail thrown away, better preserved details, and larger filesize.
For VBR, 2-pass MPEG2 encodes, a well-chosen qmat will have a net result of slightly better visual quality (lower Q) and will not affect filesize, as compared to the default/stock qmat.
This is because the qmat will help the encoder make more intelligent decisions for where to allocate bits to preserve as much detail as possible. For 2-pass VBR, different qmats will *not* affect filesize, because 2-pass VBR at a given bitrate is specifically intended to produce a predictable filesize. The well-chosen qmat will help the encoder save bits on the scenes/frames which do not need as much detail preserved, which in turn will be given to other scenes/frames which do need more bits to faithfully preserve the details, hence slightly better visual quality / lower average Q.
However, different qmats for CQ encodes *will* produce different filesizes for the same source, because different qmats will compress the source more efficiently. -
This verbiage appears to be correct. Are you trying to write a college paper?
Or is there a question in there?
I encoded 5000 frames about 12 times (12 different matrices)
2-pass VBR, and used the same bitrate each time. I used the HCenc log to see the average
quantizer on each encode.
Here's my results.
1 avamat6 2.572
2 qlb 3.491
3 MPEG 3.502
4 soulhunter 3.675
5 hclow 3.775
6 manono3 3.921
7 manono2 4.264
8 manono1 4.394
9 fox3 4.421
10 dave 5.580
11 fox2 6.046
12 fox1 6.791
The matrices at the top give the lowest Q. for a given bitrate, or the smallest files for a given CQ encode.
That's all there is to it. You may notice other things, if you look carefully. For example, the manono matrices
appear to sharpen the image slightly, while fox1 and fox2 appear to preserve the grain of the source better.
etc,etc. -
Yes, but the matrix chosen determines a base level of detail, or how much low and high frequency detail is thrown away even at the lowest Qs. And that's why the Standard Matrix (so-called by CCE and used by most encoders, but called MPEG by HCEnc), isn't worth using except in some instances where the material doesn't compress well, or too much is being put on the DVD. If using the Standard Matrix and if the resulting Q is low, you can get better quality results using a better quality matrix, better quality meaning less low frequency data thrown away by default resulting in a clearer sharper picture.
Start checking high quality Hollywood and Criterion DVDs and collect the matrices used. Do this with DGIndex set for Options->Log Quant Matrices. Then compare the numbers with those in the Standard Matrix. That doesn't necessarily mean you should use those same matrices when you're shrinking the movie by up to 50%, but only that they're using much better quality matrices to begin with. You have to fit the matrix with the material, but I almost never find myself having to use the Standard Matrix. One easy way out if using CCE is to allow it to use adaptive Q-matrix switching'. I believe HCEnc has something similar (not sure though). Then the matrix gets adjusted depending on the complexity of the material and the average bitrate (meaning mostly the numbers get halved or quartered with the less complex material). You'll find matrix switching with many modern well-made DVDs too. Maybe those use a professional CCE encoder, don't know. -
You can generate optimal matrices with http://vision.arc.nasa.gov/dctune/ - however you need to understand idea (i.e. each picture can have optimal matrix which can be suboptimal to different picture)
btw each GOP can have own matrix - feel free to automatically (manually?) generate optimal matrix for each GOP.
http://forum.doom9.org/showthread.php?t=75416 -
Thanks everyone for your help. So far, my sources look best with manono's #7 as he recommended.
I'm really pleased with everything, except with one of my samples (attached). The video starts black, then adds white text, then quickly dissolves to full image. During these first transitional frames, quality *tanks*. As soon as the image is fully dissolved, quality is excellent. I've tinkered with everything and can't seem to figure out why (disabled custom qmat, increased bitrate, messed with other settings). On other sources that don't start this way, the quality is very good. Thoughts?
Code:#!/bin/bash SOURCE=$1 DESTINATION=$2 VB=$3 VF="scale=720:480,\ hqdn3d=2:1:2,\ pp=ac" INTER_MATRIX="\ 08,08,08,09,11,13,14,17,\ 08,08,09,11,13,13,14,17,\ 08,08,11,12,13,14,17,94,\ 09,11,13,13,14,17,17,94,\ 11,11,13,13,14,17,94,94,\ 13,13,14,16,17,20,94,94,\ 13,13,14,17,94,94,94,94,\ 13,14,17,94,94,94,94,94" INTRA_MATRIX="\ 12,12,13,14,15,16,22,26,\ 12,13,14,15,16,22,26,32,\ 13,14,15,16,22,26,32,41,\ 14,15,16,22,26,32,41,53,\ 15,16,22,26,32,41,53,94,\ 16,22,26,32,41,53,70,94,\ 22,26,32,41,53,70,94,94,\ 26,32,41,53,94,94,94,94" #,Pass,1 ffmpeg \ -i $SOURCE \ -framerate 30000/1001 \ -vf "$VF" \ -aspect 16:9 \ -inter_matrix $INTER_MATRIX \ -intra_matrix $INTRA_MATRIX \ -dc 10 \ -g 12 \ -b_strategy 2 \ -brd_scale 2 \ -bf 2 \ -lmin 0 \ -lmax '320*QP2LAMBDA' \ -mblmin 1 \ -qmin 1 \ -qmax 31 \ -maxrate 9800k \ -b:v $VB \ -bf 2 \ -pre_dia_size 5 \ -dia_size 5 \ -qcomp 0.7 \ -qblur 0 \ -preme 1 \ -me_method dia \ -sc_threshold 0 \ -sc_factor 4 \ -bidir_refine 4 \ -profile:v 4 \ -mbd rd \ -mbcmp satd \ -precmp satd \ -cmp satd \ -subcmp satd \ -skipcmp satd \ -pass 1 \ -passlogfile ffmpeg \ -threads 4 \ -an \ -f mpeg2video \ -y \ /dev/null # Pass 2 ffmpeg \ -i $SOURCE \ -vf "$VF" \ -framerate 30000/1001 \ -aspect 16:9 \ -inter_matrix $INTER_MATRIX \ -intra_matrix $INTRA_MATRIX \ -dc 10 \ -g 12 \ -brd_scale 2 \ -bf 2 \ -lmin 0 \ -lmax '320*QP2LAMBDA' \ -mblmin 1 \ -qmin 1 \ -qmax 31 \ -maxrate 9800k \ -b:v $VB \ -bf 2 \ -pre_dia_size 5 \ -dia_size 5 \ -qcomp 0.7 \ -qblur 0 \ -preme 1 \ -me_method dia \ -sc_threshold 0 \ -sc_factor 4 \ -bidir_refine 4 \ -profile:v 4 \ -mbd rd \ -mbcmp satd \ -precmp satd \ -cmp satd \ -subcmp satd \ -skipcmp satd \ -pass 2 \ -passlogfile ffmpeg \ -threads 4 \ -an \ -y \ -f mpeg2video \ $DESTINATION
-
Bad scene detection - perhaps you should manually insert I frame at transition, fading are generally coded quite poor in MPEG-2 (this problem at least partially was addressed in H.264).
-
pandy is right --- you should set "-sc_threshold" to a negative value ( try something between -10000 and -40000).
HCenc has the option to change the quantization matrix during the fade-in and fade-out sequences.
P.S.: keep in mind that the avcodec compressors (probably, still) share the following design flaw --- if you use B-frames, all GOPs will be open, unless you disable the scene detection algorithm (i.e., set sc_threshold to its highest positive value). If you intend to author a DVD with functional chapters points, closed GOPs are required by many standalone DVD players.Last edited by El Heggunte; 4th Mar 2014 at 07:50.
-
Poor bitrate distribution. According to Bitrate Viewer the bitrate during that fade-in is under 1000 kbps.
<edit>
I reconstructed a cleaner version of your video with this script:
Code:Mpeg2Source("sample2-qmat7-4000k.d2v", CPU=6, Info=3) p1=Trim(0,130) p2=Trim(131,131).Loop(29,0,0).Fadeout(29) p3=Trim(191,220).Reverse().FadeIn(30) p4=Trim(192,0) p1+p2+p3+p4
The vertical black line marks the critical fade-from-black section. You can see that HcEnc allocated a lot of bitrate to that fade-in, giving it over 7000 kbps. Your encoding did the opposite, giving it less than 1000 kbps. Sample attached.
You need to check your encoder settings.Last edited by jagabo; 4th Mar 2014 at 11:21.
-
On break, just dropping by to say that I got a significantly better fade in by changing lmax from '320*QP2LAMBDA' to '21*QP2LAMBDA', and qmax from 31 to 7. I will further review your responses later.
-
Interesting. I had my eye on -sc_threshold, but didn't have any perspective on what a good value is. I will try your suggestion.
P.S.: keep in mind that the avcodec compressors (probably, still) share the following design flaw --- if you use B-frames, all GOPs will be open, unless you disable the scene detection algorithm (i.e., set sc_threshold to its highest positive value). If you intend to author a DVD with functional chapters points, closed GOPs are required by many standalone DVD players.
I'm not sure I need/want b-frames, the ffmpeg invocation you see is mostly based on a doom9 thread I read from the authors of avs2dvd, I assumed they probably know more about ffmpeg and mpeg2 for DVD than I do. Given that I'm using avcodec, do you suggest I leave out b-frames? If so, how does this translate into a changed ffmpeg invocation? (really, I'm just interested in any of your suggestions, b-frames or otherwise). Thanks again. -
-
I always used open GOPs and have never had any trouble with chapter settings, or had any problems with the chapter working properly, or playing DVDs on any standalones. Of course, chapter points can be set only on I-Frames.
What shows whether GOPs are open or closed? My understanding is closed GOPs are required only on DVDs with angles. Any DVD player that has trouble with open GOPs isn't compliant with the specs. I believe open GOPs are the norm. -
-
Well, I didn't say (and I didn't mean) that an MPEG-2 stream with open GOPs will not be played correctly (or "at all") on certain SAPs. I meant, certain SAPs will "land" on the wrong frame (so to speak =^.^= ) when you try to jump to a chapter point in an open GOP. And yes, I've seen many devices which have the "feature"
, probably because I live in a 5th-world country
As for ffmpeg, well I have to admit, I am no expert, and my knowledge about its idiocies might be somewhat outdated
I had completely forgotten about the cgop flag, but I recall it didn't work by itself (because it depended on sc_threshold="infinite"
).
Also, I should have spotted the bad choice "qmax=31", and I didn'tProbably because when I last experimented with ffmpeg's encoders, the devs had just started to royally mess with the old command-line syntax
Since then, I use ffmpeg only as a muxer and demuxer.
Last edited by El Heggunte; 4th Mar 2014 at 15:47. Reason: ...
-
I apologize if I misread, but it sure sounded like it to me:
You'll get functional chapter points using either open or closed GOPs. But now you seem to be saying you might not get them where you set them or where you want:
I meant, certain SAPs will "land" on the wrong frame (so to speak =^.^= ) when you try to jump to a chapter point in an open GOP. -
manono you have a great point, I really should pay MUCH more attention to what I write before pushing the "Post" button
If you are correct, then I have been fooled by DVD-lab PRO 2.24 all this time
I mean, whenever it told me it was setting a chapter point on an I-frame of an open-GOP stream,
it actually set it elsewhere -
I carried this thread a little offtopic once I got the right qmat. I intended to create a different thread to address tuning my other options:
https://forum.videohelp.com/threads/362861-An-exercise-in-high-quality-DVD-conversion-w...mpeg-revisited -
^ Not a problem, off-topic not always means useless
And yes, I do need more buckets of
Similar Threads
-
Adaptive Quantization
By donkeykong in forum Newbie / General discussionsReplies: 1Last Post: 5th Oct 2011, 07:23 -
XviD Quantization type, H.263 vs MPEG; and GMC for hand recorded material
By carlaron in forum Video ConversionReplies: 41Last Post: 2nd Mar 2010, 22:36 -
Need a set of hi-res custom matrices for 90 - 110 min avi's...(details)
By bxmbxm in forum SVCD2DVD & VOB2MPGReplies: 0Last Post: 4th Jan 2010, 17:18 -
HD mpeg2 transport stream re-quantization?
By live4ever in forum DVB / IPTVReplies: 2Last Post: 21st Jul 2009, 19:25 -
Which Quantization (Quantizer) Restrictions Number better? use 1 or 2
By afcoff in forum Video ConversionReplies: 2Last Post: 24th Apr 2009, 07:17