ffmpeg non linear quantizer

[dannyboy48888]

After digging around with ffmpeg and doing "ffmpeg --help encoder=mpeg2video" i stumbled upon:

-non_linear_quant <boolean> E..V....... Use nonlinear quantizer. (default false)

after testing it out I am pleased with the quality improvement in some areas for the similar bitrate like darker background gradients. Aside from the full scale not being implemented (up to q 27 is) is there any reason not to use this? the resulting mpeg-2 stream authors just fine with dvdstyler and plays on my set top players

my example 2 pass script is

pass 1:
ffmpeg -i /home/w/Desktop/video/Braveheart-1995/Braveheart-1995-raw.mkv -target ntsc-dvd -vf setfield=tff -aspect 16/9 -c:v mpeg2video -dc 10 -g 18 -maxrate 8000k -non_linear_quant true -qmax 27 -bf 2 -mbd rd -intra_matrix 8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19, 22,26,27,29,34,34,38,22,22,26,27,29,34,37,40,22,26 ,27,29,32,35,40,48,26,27,29,32,35,40,48,58,26,27,2 9,34,38,46,56,69,27,29,35,38,46,56,69,83 -inter_matrix 16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16 ,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,1 6,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16, 16,16,16,16,16,16,16,16,16,16,16,16,16,16 -flags +ilme+ildct -alternate_scan 1 -top 1 -passlogfile /home/w/Desktop/video/Braveheart-1995/Braveheart-1995-raw.passlog -threads 1 -q:v 4 -pass 1 /home/w/Desktop/video/Braveheart-1995/Braveheart-1995-raw-pass1-nonlin.m2v

pass 2 with bitrate from pass 1:

ffmpeg -i /home/w/Desktop/video/Braveheart-1995/Braveheart-1995-raw.mkv -target ntsc-dvd -vf setfield=tff -aspect 16/9 -c:v mpeg2video -non_linear_quant true -qmax 27 -dc 10 -g 18 -maxrate 8000k -bf 2 -mbd rd -intra_matrix 8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19, 22,26,27,29,34,34,38,22,22,26,27,29,34,37,40,22,26 ,27,29,32,35,40,48,26,27,29,32,35,40,48,58,26,27,2 9,34,38,46,56,69,27,29,35,38,46,56,69,83 -inter_matrix 16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16 ,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,1 6,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16, 16,16,16,16,16,16,16,16,16,16,16,16,16,16 -flags +ilme+ildct -alternate_scan 1 -top 1 -passlogfile /home/w/Desktop/video/Braveheart-1995/Braveheart-1995-raw.passlog -threads 1 -b:v 3520k -pass 2 /home/wDesktop/video/Braveheart-1995/Braveheart-1995-pass2-nonlin.m2v

This isn't a "is ffmpeg better than HCenc etc" question, it's is there something broken I haven't encountered yet as I am using Linux (Debian 12) and it would be nice to be able to encode natively instead of using another encoder via wine. ffmpeg's linear quant aint bad just this seems to preserve gradient's a bit better in my limited testing.

[pandy]

There is nothing wrong, if i recall correctly non-linear quantiser was standard (default) and you could turn it off in ancient times.

After some digging seem i was correct:
https://forum.videohelp.com/threads/50445-CCE-linear-quantizer-vs-non-linear

[dannyboy48888]

Thanks Pandy . After turning down the bitrate/raising the average Q I can see why Linear is the default. Non-linear tried to keep more detail and at quant 6 or so starts to macroblock more. Linear has a nose filter effect and will Saturate quicker so will give a better viewing experience at lower bitrates. Downside is it can't keep textures as good. Adding noise in vlc makes it look good though. So in my case with ffmpeg for DVD mastering non-linear as it can scale and keep detail and linear for backups as it filters but doesn't fall apart as much. In my Braveheart case in 2 pass linear at Q2 came out to 3.7gb and non linear Q4 was 4.5gb. the nonlinear definitely looked better gradient wise in darker scenes. When I lowered the nonlinear bitrate to match the linear looked better. I'm sure non linear can behave at higher q with some tuning, just testing is all.

[pandy]

You should familiarize with this thread https://forum.doom9.org/showthread.php?t=174620 and perhaps others - few years ago ffmpeg was subject of changes in terms of quality and bitrate control so never methods may give better results also quantization matrices may help you achieve higher quality within same bitrate budget.

[dannyboy48888]

Thank you. I'll do some more testing and won't bet the farm on ffmpeg. it does indeed seem to be a rate control issue. In hcenc for a given bitrate it seems to try to keep the visual quality uniform throughout. while ffmpeg varies it a bit more. I have the toolchain set up for it just was tinkering to see if I could get a native Linux chain is all.

[pandy]

I think you can use a starting point scripts from Manolito and others in doom9 thread.
New quality control use below listed parameters:

@REM Maximum Bitrate (maxrate) (Use 9800k max for DVD compatibility)
set VBV_MBR=9800k

@REM Maximum buffer size (bufsize) (Use 1835k max for DVD compatibility)
set VBV_MBS=1835k

@REM Fixed quality VBR (q) (Fishman0919's suggestion)
set E_FQ=2.0

@REM Minimum quantizer (qmin) (2 is good value for low bitrate, b<=1800k)
set E_MQ=1.0

@REM Minimum frame-level Lagrange multiplier (lmin) [0.01;255.0] (Fishman0919 suggests a value of 0.75)
set E_MLM=0.75

@REM Minimum macroblock-level Lagrange multiplier (mblmin) [0.01;255.0] (Fishman0919 suggests a value of 50.0)
set E_MBL=50.0

@REM Quantizer variability (qcomp) [0.00;1.00] (0.75 is good value) (Fishman0919 suggests a value of 0.7)
set E_VQ=0.70

[dannyboy48888]

Thank you, I'll review that. I was mainly curious about why ffmpeg doesn't default to non-linear quant when every modern encoder uses it. I see it's because it's rate control ain't tuned compared to it's linear quant rate control that can be tuned via everyones suggestions and get good results.

[dannyboy48888]

Well after some testing I found non-linear quant and mbd rd don't play well together at higher quants in ffmpeg. Eliminated it and all is well. with the standard mpeg matrix ffmpeg's q 5 is roughly equal to HCenc Q6 size and quality wise with AQ and lumgain disabled. File size making a 2 pass based of the 1st pass Q setting HCenc: 816.4Mb, ffmpeg: 754.2Mb . Another reason I wanted this route is ffmpeg can use the first pass in CQ mode to write the log for the 2nd pass. in other encoders I gotta do a first pass to get a idea of the size then a 2nd and 3rd using that given bitrate.

Script for ffmpeg: (I disable pass 2 then enable it after pluging in the bitate)

ffmpeg -i /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.mkv -target ntsc-dvd -vf setfield=tff -aspect 16/9 -c:v mpeg2video -dc 10 -g 18 -maxrate 8000k -non_linear_quant true -qmax 27 -bf 2 -intra_matrix 8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19, 22,26,27,29,34,34,38,22,22,26,27,29,34,37,40,22,26 ,27,29,32,35,40,48,26,27,29,32,35,40,48,58,26,27,2 9,34,38,46,56,69,27,29,35,38,46,56,69,83 -inter_matrix 16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16 ,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,1 6,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16, 16,16,16,16,16,16,16,16,16,16,16,16,16,16 -flags +ilme+ildct -alternate_scan 1 -top 1 -passlogfile /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.passlog -threads 1 -q:v 5 -pass 1 /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw-pass1.m2v
#ffmpeg -i /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.mkv -target ntsc-dvd -vf setfield=tff -aspect 16/9 -c:v mpeg2video -dc 10 -g 18 -maxrate 8000k -non_linear_quant true -qmax 27 -bf 2 -intra_matrix 8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19, 22,26,27,29,34,34,38,22,22,26,27,29,34,37,40,22,26 ,27,29,32,35,40,48,26,27,29,32,35,40,48,58,26,27,2 9,34,38,46,56,69,27,29,35,38,46,56,69,83 -inter_matrix 16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16 ,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,1 6,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16, 16,16,16,16,16,16,16,16,16,16,16,16,16,16 -flags +ilme+ildct -alternate_scan 1 -top 1 -passlogfile /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.passlog -threads 1 -b:v 2053k -pass 2 /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.m2v

HCenc settings:

*INFILE z:\home\w\desktop\video\around_the_world_in_80_day s-miniseries-2022\e01_video01.avs
*OUTFILE Z:\home\w\Desktop\video\Around_The_World_In_80_Day s-Miniseries-2022\E01_Video01-hc-q6.m2v
*BITRATE 2297
*MAXBITRATE 8000
*FRAMES 0 89462
*PROFILE best
*AUTOGOP 18
*AQ 0
*DC_PREC 10
*INTERLACED
*TFF
*CHROMADOWNSAMPLE 1
*LOSSLESS
*INTRAVLC 2
*MATRIX mpegstd
*PRIORITY normal


I'm sure there's plenty of room for improvement. my main goal was simply a mostly linux solution (I use avisynth and wine to make a raw source file first) that scales from backup quality to master quality by simply changing the Q setting in the first pass. I feel I have done this. Attached are some very non scientific captures showing the results. Thanks everyone for the help.

[pandy]

Allow me to repeat once more at some point to control quality lagrange multiplier was introduced in ffmpeg scripts - https://www.researchgate.net/publication/221279985_Lagrange_multiplier_selection_for_v...g_quantization

minimum MB Lagrange multiplier - mb_lmin
maximum MB Lagrange multiplier - mb_lmax
minimum Lagrange multipler - lmin
maximum Lagrange multipler - lmax

People reporting increased encoding quality of ffmpeg 9otherwise it was considered as quite poor) - i have impression that proper usage may be beneficial for quality encoding.

[dannyboy48888]

Fair enough Pandy. I'm pretty happy with the results that I dont feel the need to tune it more. To me at least I'm getting similar to hcenc quality at similar bitrates at at times less bitrate now.

[pandy]

I mean if this is fine for you then it is also fine for me but from pure curiosity it can be interesting experience to check if by adding this:

-lmin 0.75 -mblmin 50

you will get any difference, also i would explicitly add vbv buffer size, packet size and maxbitrate if your goal is is dvd (or at least i will add to command line '-report' and verify if proper settings are applied.

btw if your source is progressive i would disable alternate scan and ilme idct - they are for true interlace content where there is difference between fields but in progressive content they provide lower coding gain.

And last from me (i promise)

This one is also interesting http://forum.doom9.net/showthread.php?p=1965527

for example this one:

-cmp sad -subcmp satd -mbcmp rd -ildctcmp rd -dia_size 800 -precmp sad -pre_dia_size 800 -me_range 512 -mbd rd -trellis 1

[dannyboy48888]

Thanks for the input Pandy. My source is indeed TFF hard interlaced hence my flags (why is a whole other topic lol). The -target ntsc-dvd flags correctly sets the buffer, packet and max bitrate. I verified this bitrateview and MPEGValidator. for a few weeks before trying mpeg-2 non-linear I tried trellis and it is indeed broken for mpeg-2 encoding. tried -dia_size and pre_dia_size up to 1025 (exaustive) and had *very* little benefit in MSU with SSIM and VMAF tests. same with cmp etc up to 6: i got a *little* nicer picture on still shots but nothing on actual playback. -mbd rd is broken with non-linear quant. frame tearing etc happens on any q above 4 in my testing. I did a run with your suggested "-lmin 0.75 -mblmin 50". at my q 5 it did nothing and may of hurt some scenes. at Q 2 or 3 i can see it benefiting as it does allow the quant to go lower in certain scenes. really shouldn't be needed though as non-linear q 2 is pretty much linear Q1 . Why am i stuck on non-linear you may ask? simple: it has a steady curve for quality instead of a drastic change between 2-3-4 quants. with linear Q2 to Q3 isa 30% drop so either the frame get too much Q or not enough. with non-linear each Q is ~13% so quality can be evenly done. Far as why mpeg-2? I want 1 file that will play on a roku, google tv device, tablet, dvd player. dvd player via usb and PC. Attached is proper grabs from Avspmod showing the raw then encoded frames. per your suggestion the script i did was:


ffmpeg -i /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.mkv -target ntsc-dvd -vf setfield=tff -aspect 16/9 -c:v mpeg2video -dc 10 -g 18 -lmin 0.75 -mblmin 50 -non_linear_quant true -qmax 27 -bf 2 -intra_matrix 8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19, 22,26,27,29,34,34,38,22,22,26,27,29,34,37,40,22,26 ,27,29,32,35,40,48,26,27,29,32,35,40,48,58,26,27,2 9,34,38,46,56,69,27,29,35,38,46,56,69,83 -inter_matrix 16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16 ,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,1 6,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16, 16,16,16,16,16,16,16,16,16,16,16,16,16,16 -flags +ilme+ildct -alternate_scan 1 -top 1 -passlogfile /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.passlog -threads 1 -q:v 5 -pass 1 /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw-pass1-lmin.m2v
#ffmpeg -i /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.mkv -target ntsc-dvd -vf setfield=tff -aspect 16/9 -c:v mpeg2video -dc 10 -g 18 -lmin 0.75 -mblmin 50 -non_linear_quant true -qmax 27 -bf 2 -intra_matrix 8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19, 22,26,27,29,34,34,38,22,22,26,27,29,34,37,40,22,26 ,27,29,32,35,40,48,26,27,29,32,35,40,48,58,26,27,2 9,34,38,46,56,69,27,29,35,38,46,56,69,83 -inter_matrix 16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16 ,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,1 6,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16, 16,16,16,16,16,16,16,16,16,16,16,16,16,16 -flags +ilme+ildct -alternate_scan 1 -top 1 -passlogfile /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw.passlog -threads 1 -b:v 2053k -pass 2 /home/witzel/Desktop/video/Around_The_World_In_80_Days-Miniseries-2022/E01_Video01-raw-lmin.m2v

final sizes:
ffmpeg q5: 754Mb (2120Kb)
ffmpeg q5-mbl: 754Mb (2120Kb)
Hcenc Q6: 816Mb (2294Kb)

considering its 720x480 29.97i I say not too shabby quality wise vs the bitrates used.

For "mastering" I set Q to 3 in pass 1. If it's too big for a disc I adjust the bitrate in pass 2. If not I run pass 2 with the given rate to distribute the bits. for "backup" Q 5 works for me and I run pass 2 with the result from pass 1 using the script in my prior post.

By no means do you have to promise the last one lol. Been trying this for years and I'm learning what i did for linear is not needed (I.E custom FOX matrix) with non-linear as most of it was to push stuff up the qscale where each Q didn't eat 50% of the bitrate then 25% etc. Hence my original question of why aint non-linear the default?

[pandy]

Promise is a promise. Thx for elaborated reply - appreciated.