What's up with x264 zones failing so bad?

[Mephesto]

I'm redoing an encode I did back in 2009 because I noticed that one minute in the movie takes about 50MB out of the 700. It's a scene with heavy rain, about 5000 kb/s in 480p. So I decided to zone it with a high quantizer like 30, but this didn't lower the bitrate much. A 40 quantizer didn't do much either.

Encoding only that scene itself with crf=22 made it about 50MB. Cutting that scene out directstream with Avidemux revealed it to be about the same size.

So using crf40 in a zone is the equivalent of crf22 without zoning? What the hell is going on... I've already encoded this 2-hour-movie twice and I'm sick of experimenting. It looks like I'll have to give it a ridiculously high crf like 50 or even the max 60 to get any real results.

Come to think of it, before mb-tree was out I always used a quantizer of 35-38 rather than 40 for the credits because the quality was so unacceptably low. After mb-tree I noticed the quality was bizarrely good even at 40 so 40 became the default in my settings but now I think I figured out why this is after all these years. Zoning is really bugged.

Can anyone shed some light on this?

[Selur]

Not much details about used settings, versions of the tools used, which tools are used and no sample,..
-> Solar spots?

If you don't like x264 to use high bit rates, adjust the vbv settings accordingly,...

[Mephesto]

Latest x264, and I was perfectly clear on what tool I used.

Used quantizer 36 on this zoned area in the 2-hour-encode then extracted with direct stream copy
Encoded this specific scene only with crf 32.

They are the same size despite different crfs and the second one is higher quality. Why is this?

I do want x264 to use high bit rates except for this specific scene because I don't give a shit about every speck of a raindrop being perfectly preserved. Utter waste of bitrate and gravely robs quality from the rest of the movie.

x264 wisely allocates its bits most of the time but this is one of the few times where it fails to concur with the human perception of entropy.

This isn't the only rain scene either but it's the only one that's over a minute long and features little detail except the rain itself.

[Selur]

Latest x264, and I was perfectly clear on what tool I used.

No you were not!
You only wrote that you used Avidemux for cutting and that you used x264 for the encoding.
You didn't specify if you used Avidemux for reencoding or any other tool.
You didn't specify what settings aside from using different crf values were used.
-> Wild guess: Your VBV restriction overrule your zone settings.

Wish you much luck figuring this out.

Cu Selur

[Mephesto]

If I'm using the latest x264 why does it matter what GUI I use? I use Megui... I wouldn't touch Avidemux with a dog-felching eel normally but I had no patience to use a CLI to cut that scene out.

Look at the metadata of the samples I just posted, the settings are in there. As you can see, same settings for both.

I don't have a VBV restriction, the bitrates are free to flow as they please as long as the final file size amounts to 700. What exactly is being "overruled" here? I had to input a zone of crf36 to match what would've been the bitrate of a crf32 and despite the same bitrate I get lower quality. This is the weird behavior I'm asking about.

[poisondeathray]

Works ok here on some quick tests. Post your commandlines

[Mephesto]

Megui doesn't display the zones in the commandline but here it is

program --crf 20.0 --deblock -1:-1 --bframes 16 --b-adapt 2 --ref 4  --qpmin 10 --qpmax 51 --qcomp 0.7 --rc-lookahead 250 --aq-mode 2 --me  umh --direct auto --subme 11 --partitions all --trellis 2  --no-fast-pskip --no-psy --output "output" "input"

What do you mean by it works ok? Have you compared it to anything?

[poisondeathray]

I meant it looks ok in that comparing with vs. without the zone settings; the lower quality zone version had a smaller filesize with deteriorated zone...

But it looks like the "quick" tests were too quick... It might be content related - I'm testing on noise clips - (e.g. a pure noise clip will behave differently than a smooth cartoon) but there seems to be an issue where only the 1st frame or few frames in the zone are really affected in crf mode (deteriorated, as expected by the higher crf value in the zone), not sure about the other rate control methods. I'll have a closer look later, but you're right something is amiss

[Mephesto]

If you're doing a quick test I suggest you have 500 non-complex frames before and after the complex scene because I tried to do a quick test of encoding the rainy clip with crf18 but making all frames except first one a zone of crf40. It didn't work well, the clip ended up what you'd expect from using crf40 without a zone.

This isn't that big of a deal now that I know to increase the quantizer a few notches higher than what I'd normally use but I thought I'd see if it's just me or if it's the norm.

The only thing I do know is that the quantizer can't go from 18 to 40 right away but 4 quantizer notches per frame, so it takes about 6 frames for the quality of the zone to take effect but that is not relevant to this anomaly we've now seen. 12 frames don't make 1000 kb/s worth of differnece.

[poisondeathray]

For the quick test I used a few hundred before, a few hundred for very "noisy" , then a few hundred after

It's almost the same result in ABR and 2pass mode. I even tried -q and -b modes for the zone (instead of CRF)

But the 1st frame in the zone is definitely affected

EDIT: AAAHHH SHIT, I got the zone numbers wrong (I did the 1st section instead of the middle). No wonder the 1st frame was affected. Maybe it does work ok LOL . I'll report back.

[poisondeathray]

Haha I feel stoopid LOL . I was going WTF, the bitrate graph is all screwed up

It works ok. I'm 100% certain now . Stream analyzers, avg frame quantizers, bitrate graph all concur

Post your megui log , perhaps your zone arguments aren't getting passed

[_Al_]

Trying zones in megui did not work for me also, I had to use command line. I did not use CRF just multiple bitrate choice but I experienced weird result or nothing at all, not sure what, it did not work right in Megui.

[poisondeathray]

If it's true , maybe you should report zones aren't working in megui . It might also a be content related issue

Please post the logfile

[El Heggunte]

I use Megui...

.......

[_Al_]

I just test it again, it must be broken that feature somehow in Megui,or I do not know how to do it right, also possible:

Megui: I used CRF 18 for video, and used zone: start frame 300, end frame 600, weight 200% (or other test with quantizer 10), Add, OK - after render it did nothing at all , both instances, comparing it to encode with no zones set

X264: I used CRF 18: --zones 300,600,b=2 (or other test --zones 300,600,crf=10) - and it worked in both instances

[poisondeathray]

Can you guys post your megui logfiles....

In Mephesto's case, he says it does something, just the effect is not as expected

In _Al_'s case it does nothing ?

[poisondeathray]

Ok I can confirm _Al_'s findings

The x264.exe binary that comes with megui is fine (using zones with it via CL works).

The log file says the --zones option is passed, but it clearly it's not affecting the encode when used via the GUI

If you want to improve megui someone should file a bug report

[hello_hello]

I just gave MeGUI a quick spin running a small encode while specifying zones and then again without. The effect of using zones on quality was obvious and the file size changed quite a bit.

From MediaInfo after specifying zones:

Encoding settings : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=62500 / vbv_bufsize=78125 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00 / zones=0,1430,q=26/2107,4180,q=51


How is everybody adding zones to an encoding job? Via the Zones button in the video encoding section? It'll only let you use two methods when creating zones, quantizer and weight (both seem to work for me). Mephesto mentioned specifying a CRF value for a zone. I assume that'd have to be manually added to the command line in the x264 encoder configuration. I haven't tried it that way myself yet.

PS Are we all aware MeGUI automatically clears any specified zones when a script is loaded into the video section for encoding (I assume in order to not accidentally re-use them)?

PPS If it makes a difference I'm using MeGUI version 2435 (development update server). I couldn't find any recent changes or fixes to the zones function in the changelog though.

[hello_hello]

Megui doesn't display the zones in the commandline but here it is.

It should if they're added to the encoder configuration using the custom command line. How are you adding them? There's no other way to add them to the actual encoder configuration is there?

Here's the commandline from my MeGUI log file after using MeGUI's Zones button in the video encoding section (the encode in my previous post). I added some colour to make it pretty.

"Job commandline: "C:\Program Files\MeGUI\tools\x264\x264.exe" --level 4.1 --crf 18.0 --vbv-bufsize 78125 --vbv-maxrate 62500
--zones 0,1430,q=26/2107,4180,q=51 --colormatrix bt470bg --sar 1:1 --output "D:\test.mkv" "D:\test.avs" "

Next encode after adding zones to the custom command line in the encoder configuration. Using a CRF value in zones seems to work that way.

I added this:

--zones 0,1430,crf=26/2107,4180,crf=51



The log file says this:

custom command line: --zones 0,1430,crf=26/2107,4180,crf=51
--[Information] [28/12/13 6:35:48 PM] Job commandline: "C:\Program Files\MeGUI\tools\x264\x264.exe" --level 4.1 --crf 18.0 --vbv-bufsize 78125 --vbv-maxrate 62500 --colormatrix bt470bg
--zones 0,1430,crf=26/2107,4180,crf=51 --sar 1:1 --output "D:\test.mkv" "D:\test.avs"

MediaInfo displays this:

Encoding settings : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=62500 / vbv_bufsize=78125 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00 / zones=0,1430,crf=26/2107,4180,crf=51

And because this thread already contains screenshots of bitrate viewer, here's how the encode looked (crf=51 or q=51 both resulted in similar low quality).

Before the second zone:



A few frames into the second zone:



At least now I know how to add them. Maybe one day I'll want to. I've not used zones before.

[Mephesto]

Megui log:

[Information] [12/27/2013 5:54:28 PM] Job  commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2  --bitrate 700 --stats "M:\Sin City.stats" --deblock -1:-1 --keyint 240  --bframes 16 --b-adapt 2 --ref 8 --qpmin 10 --qpmax 51 --qcomp 0.7  --rc-lookahead 250 --aq-mode 2 --me umh --direct auto --subme 11  --partitions all --trellis 2 --no-fast-pskip --no-psy --zones  43624,45624,q=36/169263,178421,q=40 --sar 1:1 --output "M:\Sin City.264"  "M:\Sin City.avs"

I never questioned before whether CRF or QP is used in zones. In my zoned vs no zoned test I used CRF32 to get the 13.5MB output and Q=36 for the zoned one.

[hello_hello]

I never questioned before whether CRF or QP is used in zones. In my zoned vs no zoned test I used CRF32 to get the 13.5MB output and Q=36 for the zoned one.

I guess your many references to CRF in zones must have somehow fooled me into thinking you were referring to using a CRF value in a zone. It's probably why I was having trouble following what you were doing.

So using crf40 in a zone is the equivalent of crf22 without zoning? What the hell is going on... It looks like I'll have to give it a ridiculously high crf like 50 or even the max 60 to get any real results.

They are the same size despite different crfs and the second one is higher quality. Why is this?

I had to input a zone of crf36 to match what would've been the bitrate of a crf32 and despite the same bitrate I get lower quality. This is the weird behavior I'm asking about.

I tried to do a quick test of encoding the rainy clip with crf18 but making all frames except first one a zone of crf40.

Anyway....

In my zoned vs no zoned test I used CRF32 to get the 13.5MB output and Q=36 for the zoned one.

I just re-encoded the video I was using for testing. No zones. CQ23 the first time. CRF23 the second time. I can confirm you don't get the same bitrate when using the same value for CQ and CRF. I didn't really expect to. The former gave me an average bitrate of 814kbps. For the latter it was 587kbps. So maybe I'd need to use something like CQ30 or even CQ40 just to end up with a similar bitrate to CRF23. And it'd possibly look like crap. Maybe just like the example you referred to in your original post, aside from the fact you referred to it as CRF40 in a zone....

Why not use the weight method for zones? If you want to reduce the bitrate by 50%, wouldn't it be a simple matter of something like:
--zones 169263,178421,b=0.5
Assuming you can get MeGUI to pass it on to the encoder. I'm yet to be able to get MeGUI's zones not to work, and it seems like maybe they're working for you.... but specifying a CQ value isn't giving you the result you expected.

[poisondeathray]

I can confirm it works now through the GUI.

The megui version I had was about a year old, so something got fixed. I suggest _Al_ to update it because it seems to work ok with all methods now: with the CLI (obviously), with the "extra commandline options", and with the "zones" button

And yes, there are significant differences when using q= vs. crf= for the zone

[Mephesto]

My bad, I didn't know until now that the Q is QP and not CRF. It's been so long since anything but CRF was the default so I used the terms interchangeably.

When I saw hello_hello use crf= in his script then it got me wondering. I'll use crf= in the commandline from now on. My encode is 10% away from being complete so I'll see how it is.

I used to use QP for cartoons exclusively since it did seem to have better quality than CRF but since mb-tree came out which was only usable with CRF, this obsoleted QP.

Also, can I ask if anyone uses custom scene_cut values? I had to use a lower value because x264 kept putting I frames in the middle of each scene. No wonder the quality sucked so bad.

[poisondeathray]

Also, can I ask if anyone uses custom scene_cut values? I had to use a lower value because x264 kept putting I frames in the middle of each scene. No wonder the quality sucked so bad.

Are you sure they were from scenecut detection ? The other possibility is you've reached the specified --keyint interval . If that's the case, and quick seeking isn't an issue, consider increasing the keyint value

[_Al_]

I can confirm it works now through the GUI.

The megui version I had was about a year old, so something got fixed. ....... with the "zones" button

ok, I used v.2418, good to know it is fixed now ...

[Mephesto]

AAAAHHH damn it! I used zone crf=33 on the rainy scene because it passed a test when surrounded by 500 regular frames before and after. Now I cut that scene out from the complete movie (encoded a 4th time now) and the average bitrate of it is... 4000 kb/s. And I did this with x264.exe, no Megui invovled so this isn't a damn frontend problem.

That's it, I'll encode the complete thing one more time. Let's see this bitrate-hungry scene survive a crf=50 this time.

[hello_hello]

AAAAHHH damn it! I used zone crf=33 on the rainy scene because it passed a test when surrounded by 500 regular frames before and after. Now I cut that scene out from the complete movie (encoded a 4th time now) and the average bitrate of it is... 4000 kb/s. And I did this with x264.exe, no Megui invovled so this isn't a damn frontend problem.

That's it, I'll encode the complete thing one more time. Let's see this bitrate-hungry scene survive a crf=50 this time.

I did wonder if CRF could be used for zones in conjunction with 2 pass encoding. The x264 help file doesn't say you can't, but 2 pass and CRF are trying to achieve different goals in some respects.
I still have no idea why you don't just use the weight function and reduce the bitrate that way. You know what the bitrate normally is and how much you want to reduce it. Why keep guessing at CRF values etc?

Also, can I ask if anyone uses custom scene_cut values? I had to use a lower value because x264 kept putting I frames in the middle of each scene. No wonder the quality sucked so bad.

I've always thought x264 does a pretty good job with I-frame placement, but if the video is very dark when a scene changes etc... then again the less difference there is, the less it probably matters.

Not so long ago I encoded a large number of videos (not cartoons) for which I only wanted to apply noise reduction to some sections, although quite a few sections. Often I couldn't decide exactly where use it, so in the end I encoded the lazy way.... each video with noise reduction and then again without it. Later I split both encodes and appended the wanted sections from each to give me the final video. 99% of the time, I could split the video on scene changes, so splitting and appending was nice and easy. About the only time there wasn't an I-frame on a scene change was if a scene's length happened to be a bit more than (roughly) 10 seconds long, which I assume is because --keyint 250/240 then gets involved. I don't know if any other x264 settings might effect I-frame placement. I rarely deviate from the defaults (medium or slow speed preset).
Whether 2 pass encoding effects I-frame placement I have no idea as I always use CRF encoding.

[Mephesto]

Sigh, it failed again. I set zone to crf50, crf ****in 50, something that normally renders the video unrecognizable and this time I got an increase of 10 kb/s, from 4114 to 4124 kb/s. Now, I wouldn't mind so much if it was possible to positively determine whether or not this works WITHOUT wasting 2 hours each time.

I did wonder if CRF could be used for zones in conjunction with 2 pass encoding.

I wonder too because it seems to work yet doesn't. It works when surrounded by 500 frames on both sides, doesn't work when surrounded by 90,000 frames on both sides.

I'm not guessing. I know exactly what quality to expect at what CRF value and that rain looks perfectly fine at crf32 with an acceptably low bitrate. The problem is it doesn't work. I don't expect any different from weight but I'll try that next.

I've always thought x264 does a pretty good job with I-frame placement, but if the video is very dark when a scene changes etc... then again the less difference there is, the less it probably matters.

It does a pretty good job when my release group encoder operates it instead of me. His workflow is always flawless. It's Christmas break and whenever he's gone, x264 ****s up on me. I really should've just set a maximum allowed bitrate to avoid this altogether. I actually still have the old 2009 rip which I had no hand in and I don't see anything special in the settings. Default scene_cut threshold, default vbv and the rain scene at least is half of 4000 kb/s.

The scenes are admittedly dark but there's an obvious sharp transition from scene to scene, far more obvious than treating the transition as a P frame and a non-transition in the middle of the scene as an I frame. Oh well, a simple tune-up fixed this.

I'll try weight now.

[Mephesto]

I don't have a lot of time. Thought I'd drop in and say that weightp worked. Thanks hello_hello

[hello_hello]

Excellent!

For CRF to work in a zone when using 2 pass encoding you'd assume the encoder would need to run the usual 1st pass, decide how much bitrate the zone was going to require, subtract that from the total specified bitrate, then use what's left for the remainder of the video. I wonder if it'd need to encode the zone using CRF, or whether it can work it out just using a normal first pass. Or maybe 2 pass and CRF in zones just won't play nice together.....