Anyone try x264's new AQ mode?

[sophisticles]

I just discovered that x264 has a new AQ mode --aq-mode 3 and I must say it really makes a significant difference in quality. I have encodes I have done with ever possible combination of settings I could think of and I was never pleased with the results but I just downloaded the latest Hybrid build and as I was playing around with some custom x264 settings I noticed the new AQ setting, so I figured I would try it mated to --qcomp 0.9.

I am shocked by the results and I think from now on that these will be 2 of my default settings for my encodes. I haven't had a chance to test it on many sources, I was just wondering if anyone else has noticed this new mode, tried it and what they thought of it.

[poisondeathray]

The original --aq-mode 3 was introduced a few years ago. Some binaries (like the ones taro_06 builds) had them. It looks from the changelog that BugMaster made the commit to the main branch. So if this is indeed the same AQ patch, the net effect (from my testing) is basically a stronger mode 1 . So this can be good or bad depending on the scenario

[jagabo]

The change log says

New AQ mode: auto-variance AQ with bias to dark scenes
commit | commitdiff | tree
Anton Mitrofanov [Tue, 6 Sep 2011 18:53:29 +0100 (21:53 +0400)]
New AQ mode: auto-variance AQ with bias to dark scenes

Also known as --aq-mode 3 or auto-variance AQ modification.

Sorely needed!

[sophisticles]

It seems that it does more than "auto-variance AQ with bias to dark scenes", in my tests with say a White person wearing shorts and black boots and dark hair it results in much nicer encodes than before.

Now they need to work on some of the more pressing things on this list:

https://wiki.videolan.org/X264_TODO/#Psy

Particularly psy, perhaps they should reverse engineer Ateme's encoder, it has both macroblock and frame based psy optimizations and from what I have heard it seems to do a much better job than x264's version.

[poisondeathray]

Did you do controlled tests at the same bitrate ? (or adjust the CRF until you get the same bitrates? ) . Unless something has changed, edges tend to be worse , at least with the old --aq-mode 3 . It might not be visible if you were using high bitrates. I think this is one of the the main reason it wasn't commited a few years ago. If you were using uncontrolled CRF, likely the bitrates were higher so of course everything will look better. IMO mode 1 is still the most balanced. Mode 4 was not much different to mode 2.

[jagabo]

Did you do controlled tests at the same bitrate ?... Unless something has changed, edges tend to be worse

Yes. But for me the banding in dark scenes is much more noticeable than slightly rougher edges on moving objects.

[poisondeathray]

^ That's why there are so many options

IIRC, the main problem was with grainy movies. There was a "halo" around objects, and frame edges, unless you used higher bitrates. This was a lot worse IMO on some content than some banding in some dark scenes . On "clean" content the effect wasn't as noticable

(I'm making some assumptions that the committed --aq-mode 3 is the same as the old one. It might be they made some tweaks, I'm not even sure why they chose to commit it now ? It's been in "beta" status for about 5 years )