Help smoothing blocky edges

[Stevie Hyper]

I have been using megui for my encoding of ts captures from the tv, I have been really struggling to reduce/remove banding and dancing pixels in darker areas on the final conversion. Here are two screenshots
[Attachment 31488 - Click to enlarge]
[Attachment 31489 - Click to enlarge] its not that easy to see but im producing hard blocky edges and need some advice on how to reduce them.

[jagabo]

If you're using noise reduction filters -- stop using them. Noise is what keeps posterization artifacts like this from showing up. If they're already in your source you can use an avisynth filter like gradfun2d or gradfun3. But you must use sufficient bitrate when encoding to keep the posterization from reappearing.

One of your images after gradfun2dbmod():

[Stevie Hyper]

Yes the original source does contain some posterization but not the amount I am producing. I have used an experimented with both gradfun2dbmod() and gradfun3() but I am unable to smooth the hard blocky edges. I also use a crf setting to 15 and no denoiser in the script so finding it hard to see why I cant achieve similar to your image?

[jagabo]

Try using the tune=grain optimization of x264. Or aq-strength=1.6 to 1.8.

[Stevie Hyper]

After a few tests the results appear a little improved but the dancing pixels and hard blocky edges are still very strong. With AQ strength raised to 1.6 I was forced to drop crf back down to 22 as the output file size was 3 times the original. is there any other ways to try or is this a problem that that I should just get used to?

[poisondeathray]

Retaining the dither is expensive to encode in terms of bitrate. If you dither the whole file and use a higher AQ it's expected to get a file several times larger.

So the other option is to use --zones to manually specify those sections that need adjusted settings. AQ isn't affected by zones, but you can use a bitrate multiplier, lowered deadzones and that will fix it without ballooning the entire filesize. You can combine that with selective filtering as well (apply filter only to some frames or sections within a frame with masks). Unfortunately this requires some user intervention - it's not simple "set and forget" because you need to preview and specify affected sections with frame numbers

[jagabo]

Since the problem is worse in dark areas you can try using an intensity mask to add grain only to dark areas:

Overlay(gradfun2dbmod(),  last,  mask=ColorYUV(cont_y=1000, off_y=350))

[Stevie Hyper]

Is this code to be placed at the end of my script? Also I have my deadzones down to 6-6, would dropping it to 0-0 have a significant affect? I haven't heard of --zones before this sounds intriguing is this part of X264 settings?

[poisondeathray]

Is this code to be placed at the end of my script? Also I have my deadzones down to 6-6, would dropping it to 0-0 have a significant affect? I haven't heard of --zones before this sounds intriguing is this part of X264 settings?

Jagabo's code is applied through the script. If you wanted to apply the filters like gradfun3() etc...to various sections those would be applied through the script. Those are filters. You could also use Trim() to divide it up into sections, or ApplyRange(), or several other techniques. Appying filters to different sections are discussed elsewhere other threads


--zones are an option for adjusting x264 encoding settings

You can think of it as applying different encoding settings for different sections. ie User override for x264's internal logic and decisions

You would do this so the entire file doesn't balloon up, only those sections that need it are improved

eg. Lets say you wanted improve a dark section with a bitrate multiplier, from frames 1000-2000 inclusive, doubling the normal bitrate in that section, and reucing the deadzones intra and inter to 6

--zones 1000,2000,b=2,deadzone-intra=6,deadzone-inter=6

In this example, frames 0-999, 2001 to end use the "normal" settings that you specified, only frames 1000-2000 use the modified settings

[newpball]

I have been using megui for my encoding of ts captures from the tv, I have been really struggling to reduce/remove banding and dancing pixels in darker areas on the final conversion. Here are two screenshots < screenshots> its not that easy to see but im producing hard blocky edges and need some advice on how to reduce them.

The first question is did you do anything that caused them in the first place? It may not matter for the current video but it will certainly help in the future to avoid those situation from happening again.

When you say ts captures from TV what exactly are you talking about and how do you capture with which codec and what setting?

[Stevie Hyper]

I use a Hauppauge pvr that I capture on to my computer with arcsoft total media extreme in mpeg4 avc 1080i at 13.5mbps, I use QTGMC or yadif to deinterlace then resize to 720p. Settings
are.... program --level 4.1 --bluray-compat --preset fast --crf 16.0 --bframes 2 --b-adapt 2 --ref 5 --slices 4 --chroma-qp-offset -2 --vbv-bufsize 30000 --vbv-maxrate 40000 --rc-lookahead 40 --aq-mode 0 --merange 24 --direct none --subme 9 --partitions all --trellis 0 --psy-rd 0:0 --no-dct-decimate --no-fast-pskip --range tv --colorprim bt709 --transfer bt709 --colormatrix bt709 --output "output" "input"