AutoMKV vs AutoGK (xvid encoding quality)

[dphirschler]

I've been using AutoGK for quite some time for my xvid encodes. I recently discovered AutoMKV and used it successfully for x264 MKV encodes. But when I used AutoMKV for xvid encodes, I cannot match the quality of AutoGK's encodes. Surely it can match and even surpass AutoGK, right?

So then, my question is, what settings can I fiddle with to achieve better quality encodes? I prefer using AutoMKV because I can give it AviSynth scripts.

FWIW, the videos I am encoding are Popeye the Sailor from DVD. Attached are screen captures for comparison. Both files were encoded at ~73MB. This cartoon is 6:04.

AutoMKV


AutoGK



Darryl

[manono]

The 1-click program did a better job? Let's see, AutoGK uses Undot on everything for a little bit of added compressibility. But it doesn't really make any kind of a visual difference.

To be sure of what AutoGK did, a log would be needed. Unless you used the ESS chipset option, it used a custom quant matrix. That might make a noticeable difference. Maybe the biggest difference is that AutoGK severely restricts the quants. This could wind up giving complex scenes better quality at the slight expense of the more-or-less static scenes, when compared to a different encode of the same video for the same size and resolution and default quant settings (1-31 or even 2-31). I've never used AutoMKV and have no idea of the settings used in it. Nor do I know how your AviSynth script is different from the one generated by AutoGK since you didn't include it.

[dphirschler]

In this case (the Popeye cartoons), I had no AviSynth script. I simply fed it the vob files from the DVD. I never thought to look into the AutoGK AviSynth script. It does make sense that it is being filtered for compressability.

I just checked AutoGK's avs file and it is doing "RemoveGrain(mode=2)", so that is probably it. I will add that to the AutoMKV options and report back.


Darryl

[dphirschler]

OK it finished encoding and although it is "smoother", it's not really much better. The blocking artifacts are still there as in the above capture. Any other ideas? How do these custom matrixes help? At the moment, AutoMKV is using "Sharktooth's EQM v3LR.cqm". I have no idea what matrix AutoGK uses.


Darryl

[dragonkeeper]

Post your settings for both. Also I suggest giving Handbrake a try, I have been satisfied with Handbrakes default settings.

[manono]

Undot is pretty much equivalent to RemoveGrain(mode=2), and neither do very much. Like I said, an AutoGK log could be useful, and I also said one major difference might be the restricted quantizers AutoGK uses. And maybe the matrix used. But since you used a fairly low bitrate matrix in AutoMKV (meaning one that smooths away some of the detail), maybe that's not the problem. Just a guess, though, in the absence of any hard information yet. Short samples of both of a scene that AutoMKV didn't handle well could also help.

[dphirschler]

OK, I think I attached the AutoGK log to this post.

As for bitrate, I am encoding these cartoons (each six to seven minutes long) so that the final size is 73MB. I also noticed that AutoGK made a slightly bigger file than AutoMKV did. AutoGK leaves the audio at 48KHz, but I am reducing it to 44.1KHz in AutoMKV, but still it makes little difference visually.


Darryl

[manono]

That's not the right log. It's an XviD Stats file.

[dphirschler]

I had a time locating a log. Still not sure this is the right thing you are looking for, but here it is. I basically captured this text right off of the log window on AutoGK. EDIT: Duh! It saves the log in the same place as the avi file. I have been automatically deleting those for so long that I forgot they were even there!

[9/3/2011 5:50:02 PM] AutoGK 2.55
[9/3/2011 5:50:02 PM] OS: WinXP (5.1.2600).2
[9/3/2011 5:50:02 PM] Job started.
[9/3/2011 5:50:02 PM] Input file: G:\video_work\Popeye\Popeye - Season 01\02 I Yam What I Yam (1933).vob
[9/3/2011 5:50:02 PM] Output file: G:\video_work\Popeye\Popeye - Season 01\02 I Yam What I Yam (1933)_AutoGK.avi
[9/3/2011 5:50:02 PM] Output codec: XviD
[9/3/2011 5:50:02 PM] Audio 1: Audio Stream 0 AC3 
[9/3/2011 5:50:02 PM] Subtitles: none
[9/3/2011 5:50:02 PM] Format: AVI
[9/3/2011 5:50:02 PM] Target size: 73Mb
[9/3/2011 5:50:02 PM] Custom resolution settings: fixed width of 640 pixels
[9/3/2011 5:50:02 PM] Audio 1 settings: VBR MP3 with average bitrate: 128Kbps
[9/3/2011 5:50:02 PM] Standalone support enabled: MTK/Sigma
[9/3/2011 5:50:02 PM] Started encoding.
[9/3/2011 5:50:02 PM] Demuxing and indexing.
[9/3/2011 5:50:04 PM] Processing file: G:\video_work\Popeye\Popeye - Season 01\02 I Yam What I Yam (1933).vob
[9/3/2011 5:50:04 PM] Source resolution: 720x480
[9/3/2011 5:50:04 PM] Found NTSC source.
[9/3/2011 5:50:04 PM] Source aspect ratio: 4:3
[9/3/2011 5:50:04 PM] Source seems to be pure FILM.
[9/3/2011 5:50:04 PM] Output will contain 8749 frames
[9/3/2011 5:50:04 PM] Decoding audio.
[9/3/2011 5:50:06 PM] Normalizing audio.
[9/3/2011 5:50:08 PM] Encoding audio.
[9/3/2011 5:50:24 PM] Using VAQ in XviD
[9/3/2011 5:50:24 PM] Audio1 size: 4,009,488 bytes (3.82 Mb)
[9/3/2011 5:50:24 PM] Overhead: 349,952 bytes (0.33 Mb)
[9/3/2011 5:50:24 PM] Video size: 72,186,608 bytes (68.84 Mb)
[9/3/2011 5:50:24 PM] Running compressibility test.
[9/3/2011 5:51:09 PM] Duration was: 44 seconds
[9/3/2011 5:51:09 PM] Speed was: 45.09 fps.
[9/3/2011 5:51:09 PM] Compressibility percentage is: 23.69
[9/3/2011 5:51:09 PM] Using softer resizer.
[9/3/2011 5:51:09 PM] Chosen resolution is: 640x480 ( AR: 1.33 )
[9/3/2011 5:51:09 PM] Predicted comptest value is: 24.57%
[9/3/2011 5:51:09 PM] Running first pass.
[9/3/2011 5:52:56 PM] Duration was: 1 minute, 47 seconds
[9/3/2011 5:52:56 PM] Speed was: 81.69 fps.
[9/3/2011 5:52:56 PM] Expected quality of first pass size: 25.38%
[9/3/2011 5:52:56 PM] Running second pass.
[9/3/2011 5:55:19 PM] Duration was: 2 minutes 22 seconds
[9/3/2011 5:55:19 PM] Speed was: 61.52 fps.
[9/3/2011 5:55:19 PM] Job finished. Total time: 5 minutes 17 seconds

Darryl

[manono]

Yeah, that's the right one. OK, the filesize chosen for the resolution you wanted was way too low. Apparently this thing doesn't compress all that well, because even after AutoGK did all it could to improve the quality, the resulting quality was still only 25.38%. That's very very low. I'd suggest a much lower resolution if you're not willing to increase the final size to 100MB or so, maybe 512x384. And I'd probably filter the hell out of it using FFT3DFilter or some such. That should get rid of your blocks and all that noise in the AutoMKV one. I don't really know why it looks so much worse. Maybe you're using a much sharper resizer, but it shouldn't make that much of a difference. Maybe the AutoGK pic is of an I-Frame and the AutoMKV pic is of a B-Frame. That could explain it. Maybe compare frames right after a scene change. They should both set I-Frames after a scene change.

You have standalone support enabled, so there aren't any custom quants used, as is usually the case. I guess it's using the H.263 matrix, but I'm not positive about that.

[dphirschler]

Here is what I know:

- Both AutoGK and AutoMKV are resizing to 640x480.
- Both AutoGK and AutoMKV are using "RemoveGrain(mode=2)"... at least now they are.
- Both AutoGK and AutoMKV are encoding for the same filesize.
- AutoGK is using BicubicResize.
- AutoMKV was using Spline36Resize.
- I am showing the same frame (b-frame or not) from both encodes. I basically load the avi file into VirtualDub and hit PgDn to go ahead 50 frames. But the entire scene looks the same, with the same blockiness (or without in the case of AutoGK). It fades in from black.

I am not willing to chance the frame size smaller than 640x480. Since I know AutoGK can do it with acceptable quality, I expect AutoMKV to be able to do it too.

So the next thing to try is changing the resize method from Spline35 to Bicubic. Apparently, a "softer" resize makes a difference in compressability.

manono, I appeciate you hanging with me on this. I am learning a lot about xvid compressability, and I hope anybody else following this thread is too. And I do think I can get to the bottom of this puzzle (with your help hopefully).


Darryl

[manono]

You might try and run both AVIs through DRF Analyzer and compare what it has to say about them. Some of the AutoGK settings are somewhat non-standard (number of B-Frames, restricted quants), and DRF Analyzer might help you in telling the differences. Also, BitrateViewer might also give you some good information about the 2 encodes.

http://www.divx-digest.com/software/drfanalyzer.html

Yes, Spline36 is a sharper resizer, but it shouldn't make all that much of a difference, I don't think.

[jagabo]

Yes, a softer resizer will reduce the bitrate requirement a bit. Bilinear will be softest. The difference between Bicubic and Spline36 isn't that great.

Also consider is the source. Does it have blocky artifacts? Use a deblocking MPEG 2 decoder (DgDecode for example) to reduce those blocks before encoding.

Do both AVI files have the same fourcc? DX50 vs. XVID for example. If they're different you might be using two different decoders -- one with deblocking enabled, one with deblocking disabled.

[dphirschler]

No meaningful difference using BicubicResize. I am attaching AutoMKV's log file. Maybe there is something useful there.


Darryl

[poisondeathray]

your automkv encode is using a different matrix , and xvidencraw (slightly different than xvidvfw, with a CLI interface)

Using Matrix: Sharktooth's EQM v3LR.cqm

BTW, the author has discontinuted automkv a few years ago, it has been replaced by hdconverttox

[jagabo]

Sharktooth's matrices:
http://forum.doom9.org/showthread.php?p=552539

[dphirschler]

Here are DRF Analyzer reports on both files:

AutoGK:

DivX DRF Analyzer v0.9.3 Report!
File Name: G:\video_work\Popeye\Popeye - Season 01\02 I Yam What I Yam (1933)_AutoGK.avi
FourCC: XVID
Codec: XviD0050
Resolution: [ Width: 640 Height: 480 ]
Frame Rate: 23.976 frames per second
The Video has 8749 frames [ 00:06:04 ]

Average Frame quality is  HIGH    [Average DRF/quantizer is 0.00]
Standard Deviation: Quality is HIGH    [Std. Deviation is 0.00]
Image Resolution is HIGH
There are NO frame drops ( NO drops is better )
This video may have some frames with VERY POOR Quality 

Recomended Resolution: [0x0]     (Target DRF/quantizer=~2.80)
The filesize should be larger! 

Performance Caracteristics: 
Macroblocks per frame: 1200 ( Poor Playback in Slow Computers, PIII450 or better required ) 
The Width is multiple of 32

Kilobits per Second: 1551.07
Kilobits per Frame: 64.53
Kilobits per Macroblock: 0.054
Bits per Pixel: 0.22

Frame Type Statistics :
I Frames: 0.86%
P Frames: 60.44%
B Frames: 38.70%
S Frames: 0.00%
N Frames: 0.00%
(More Advanced Codecs use B and S frames)

http://www.geocities.com/analyzerDRF/

AutoMKV:

DivX DRF Analyzer v0.9.3 Report!
File Name: G:\video_work\Popeye\Popeye - Season 01\131514784802 I Yam What I Yam (1933).avi
FourCC: XVID
Codec: XviD0050
Resolution: [ Width: 640 Height: 480 ]
Frame Rate: 23.976 frames per second
The Video has 8748 frames [ 00:06:04 ]

Average Frame quality is  HIGH    [Average DRF/quantizer is 0.00]
Standard Deviation: Quality is HIGH    [Std. Deviation is 0.00]
Image Resolution is HIGH
There are SOME frame drops ( NO drops is better )

Recomended Resolution: [0x0]     (Target DRF/quantizer=~2.80)
The filesize should be larger! 

Performance Caracteristics: 
Macroblocks per frame: 1200 ( Poor Playback in Slow Computers, PIII450 or better required ) 
The Width is multiple of 32

Kilobits per Second: 1519.77
Kilobits per Frame: 63.23
Kilobits per Macroblock: 0.053
Bits per Pixel: 0.21

Frame Type Statistics :
I Frames: 0.82%
P Frames: 56.96%
B Frames: 42.22%
S Frames: 0.00%
N Frames: 0.00%
(More Advanced Codecs use B and S frames)

http://www.geocities.com/analyzerDRF/

What do you guys make of this?

Oh, and here is the same frame from the DVD (resized to 640x480):


The "Popeye the Sailor (1933-1938)" set has seen high praise for the quality of restoration and its presentation on DVD. I doubt it is suffering from any blocking. The blocking is somehow being introduced in AutoMKV's xvid encoding process.


Darryl

[jagabo]

Could you provide an MPG or M2V sample of the opening credits?

[dphirschler]

OK, I am attaching a clip of the opening credits (Wow! It takes a long time for me to upload 22MB). I used Mpg2Cut2 to cut it.

I looked into HDConvertToX. It seems a bit quirky. I had trouble setting my final encoded resolution to 640x480. And I didn't see any option in there to encode as film (23.976 fps progressive). It simply reported the file as 29.97 fps, which I guess is correct if you assume 3:2 pulldown on playback. But it's encoded progressive 23.976 on the DVD.


Darryl

[dphirschler]

I have a theory. It could be some sort of post processing deinterlacing filter. I've been noticing jaggies on diagonals. I changed the "PP=6" setting to "PP=0" (no processing). Doing a test encode right now. UPDATE: Well, "PP=0" didn't have much of an effect. But I still suspect it has something to do with post processing.


Darryl

[jagabo]

Thanks for uploading the clip. I'll take a look at it tomorrow.

[poisondeathray]

Did you try disabling the cqm ?

If the only reason you're using automkv is for avisynth support with xvid encoding, there are other (almost 1 click) programs eg. megui, xvid4psp

Some reasons why it won't compress as well : the flickering (luminance changes), dirt spots, and shaking (x,y translation of the frame) . If you cleaned up the dirt, flicker and stabilized it, it would compress better . You mentioned it was restored - maybe it was left like that on purpose? Some people prefer it that way think it adds to the character of old films

[jagabo]

The video is pretty noisy but I get about the same as AutoGK when I encode with VirtualDub and Xvid at similar bitrates and the h.263 matrix (no filtering at all). I don't know why AutoMKV is doing so much worse. Try using Mpeg File Birate Viewer and comparing bitrate peaks and dips.

Just for fun:

Mpeg2Source("02 I Yam What I Yam (1933)_clip.d2v", CPU=6)
TFM(d2v="C:\Users\John\Desktop\02 I Yam What I Yam (1933)_clip.d2v")
TDecimate()
Stab()
DeFlicker(scene=30)
RemoveSpots()
Crop(2,2,-2,-2)
AddBorders(2,2,2,2)
GreyScale()

x264 CRF=20 MKV attached. Less flicker, less shake, fewer spots.

[dphirschler]

Jagabo, that looks fantastic. However, my goal here is not archival quality. I am only trying to get AutoMKV to encode xvid as well as AutoGK. It should be a simple task, but somehow despite having a plethora of settings to tweak, AutoMKV fails.

I am attaching two sample clips to this post - one encoded with AutoGK, the other with AutoMKV, both are xvid. Seeing it in motion really brings home the point, I think. Both look great, but AutoMKV has terrible blocking during the opening titles. It doesn't make sense because the scene is mostly static, except for a fade in and dissolve out. The following scene is much more difficult to compress IMO. It's super-detailed, lots of motion, rain, and lighting. Yet it looks fine... very watchable.


Darryl

[dphirschler]

Here are the "DRF" graphs of the first AutoGK and AutoMKV xvid encodes. I am not sure how to read these graphs, but I instantly notice the red section at the beginning of the AutoGK graphs and the absence of it in the AutoMKV graph. What does that mean?

AutoMKV


AutoGK



Darryl

[dphirschler]

I think I solved it! I selected the "Constant_Quality.xml" profile and "variable constant quality" so that it can match my final file size. It finally did a high quality encode from beginning to end, including the opening titles. In fact, it surpassed AutoGK! Check out the screen cap below.

AutoMKV (CQ)


This is what I was expecting all along. Some more thorough documentation would have helped me get to this point a lot quicker. But I guess I can't complain too much. It is free software, and (finally) kicks ass!

Now I am going to fiddle some more with the resize setting to see if I can get it even sharper! Currently it is using "bilinear". I want to try "spline36". But honestly, if I cannot improve beyond this, I will be okay with that.


Darryl

[jagabo]

Looking at the bitrate graphs of the two videos, it looks like AutoMKV took more bitrate away from earlier parts of the clip to concentrate on the end of the clip (the heavy rain sequence).

[dphirschler]

I think the softer resize (bilinear) is the better one unless you have a really clean source... then spline36 is better. Obviously, a 1930's era cartoon is not going to be the cleanest source.

Here is my AutoMKV xvid settings file, in case you want to give it a try. This works well on a 6-7 minute cartoon.

Save file as xvid_dph.ini

; AutoMKV parameters
[AutoMKV]
Source on DVD = 0
Input File = G:\video_work\Popeye\Popeye - Season 01\01 Popeye the Sailor (1933).vob
Input Folder = G:\video_work\Popeye\Popeye - Season 01\
Output Folder = G:\video_work\Popeye\Popeye - Season 01\
Container = AVI
Movie Name = 01 Popeye the Sailor (1933)
Size = 73
Slice = 1
Resizer = 0
Width = 640
Set encoding mode = XviD
Audio Track 1 = 80: AC3 1/0 192
Audio Codec 1 = Lame
Audio Bit 1 = 9
Audio Bitrate 1 = 128
Audio Channel 1 = Mono
Audio Track 2 = NONE
Audio Codec 2 = Lame
Audio Bit 2 = 7
Audio Bitrate 2 = 112
Audio Channel 2 = Stereo
Sub Track 1 = NONE
Sub Track 2 = NONE
Encoding Profiles = Constant_Quality.xml
Hit Media/Go Unlimited = Selected Hit Media Size
xcrfgounlimited = 0
xcrfhitmediasize = 1
xcrfcrfquantitizer = 2
Use CRF/CQ for AutoResize = 1
Auto adjust CRF = 1
AQ Strenght (x10) = 0
AQ Mode = 0
Use Last line AVS Script = 0
Last Line AVS Script = 
Save Temp Folder = 0
CRF/CQ Value = 8.26
Bitrate Encoding = 0
Bitrate Value = 700
Quality Encoding = 0
Quality Value = 75
Delete temporary files = 0
Encrust/Burn Subtitles = 0
Enable Shutdown = 0
Process this IFO = 0
X264 Psy-RDO = 0
PGC to Process = -1
Force ITU Resizing = 0
Anamorphic Encoding = 0
Filter = RemoveGrain(mode=2)
Enable Last Line AVS Script = 0
Deinterlacer = Film
Field Order = AUTO
Language = eng
Trim from = 
FPS = 0.000000
Handle Delay using = 1
Use ProjectX for TS files = 0
Use custom X.ini = 0
HQ Deinterlacer = 0
Matrix = Didees SixOfNine-HVS.cqm
Force Mux AR = 1
Force Mux W*H Width = 0
Force Mux W*H Height = 0
High Quality First Pass = 0
Do the Third Pass = 0
Specify AR input = 1.333334
--sar = 
Use DirectShowSource for Audio/Video = 0
Use DirectShowSource for Audio = 0
Enable Manual Crop And Resize = 0
Crop Top = 
Crop Left = 
Crop Bottom = 
Crop Right = 
Manual Width = 
Manual Height = 
Audio Track Name 1 = eng
Movie Title Name = 
Audio Track Desc 1 = 
Nero Profiles 1 = AUTO
Nero Encoding Mode 1 = Quality
CCT Profiles 1 = AUTO
Channel Options 1 = AUTO
MP3 Settings 1 = ABR
Windows Media Audio = AUTO
Allow Hybrid Order/Pattern = 0
Windows Media Channel = AUTO
Normalize Audio 1 = 1
Frequency 1 = 44100
Tempo 1 = NONE
Pitch 1 = NONE
OtherADD 1 = 
Audio Track Name 2 = 
Audio Track Desc 2 = 
Nero Profiles 2 = AUTO
Nero Encoding Mode 2 = Quality
CCT Profiles 2 = AUTO
Channel Options 2 = AUTO
MP3 Settings 2 = ABR
Normalize Audio 2 = 1
Frequency 2 = AUTO
Tempo 2 = NONE
Pitch 2 = NONE
OtherADD 2 = 
Nero 2 Pass Encoding 1 track = 
Nero 2 Pass Encoding 2 track = 
Allow AVS check = 0
Hide Subtitle = 0
engage keep_bitstream_ar_info = 0
Number of Threads to use = 0

Darryl