Handbrake - deblocking has opposite effect??

[starplayer]

Hi guys.
I'm having some trouble encoding a .vob file from a dvd, to x264 with handbrake.
However, no matter what settings I choose, I''m getting ugly blocking artifacts. Even with no filters and 18 CRF, they are present in some scenes.
Here's an example, as it's more visible in motion, while in the source the blocking is nowhere near as noticeable.
The weirdest thing is that, if I turn the deblocking filter on, no matter the size, instead of reducing the blocking, the image instead gets more blocky. If I increase the strenght, instead of smearing, blocks get more visible instead.
Where I notice the most is in the face of the dude on the left. Terrible blocking shimering.
I annex the cropped excerpt from the original, in case anyone wants to try encoding it without making it significantly worse than the original. Improving it in any way would be even better of course.
Notice the horizontal line under the dude nose, and on top of the woman's nose. They are not present in the original, at least they're not so jarring.


[Attachment 62747 - Click to enlarge]

[23:44:05] CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
[23:44:05]  - Intel microarchitecture Kaby Lake
[23:44:05]  - logical processor count: 12
[23:44:05] Intel Quick Sync Video support: no
[23:44:05] hb_scan: path=/media/starplayer/Sacola1/sotao/History Channel Barbarians & Barbarians 2/BARBARIANSDISC1/VIDEO_TS/VTS_01_1.VOB, title_index=1
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:333: failed opening UDF image /media/starplayer/Sacola1/sotao/History Channel Barbarians & Barbarians 2/BARBARIANSDISC1/VIDEO_TS/VTS_01_1.VOB
disc.c:437: error opening file BDMV/index.bdmv
disc.c:437: error opening file BDMV/BACKUP/index.bdmv
bluray.c:2646: nav_get_title_list(/media/starplayer/Sacola1/sotao/History Channel Barbarians & Barbarians 2/BARBARIANSDISC1/VIDEO_TS/VTS_01_1.VOB/) failed
[23:44:05] bd: not a bd - trying as a stream/file instead
libdvdread: DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav: vm: vm: failed to read VIDEO_TS.IFO
[23:44:05] dvd: not a dvd - trying as a stream/file instead
[23:44:05] file is MPEG Program Stream
[23:44:05] Probing 1 unknown stream
[23:44:05]     Probe: Found stream mpegvideo. stream id 0xe0-0x0
[23:44:05] Found the following streams
[23:44:05]     Video Streams : 
[23:44:05]       0xe0-0x0 type mpegvideo (0x2)
[23:44:05]     Audio Streams : 
[23:44:05]       0xbd-0x80 type AC3 (0x81)
[23:44:05]     Subtitle Streams : 
[23:44:05]     Other Streams : 
[23:44:05] stream id 0xbd (type 0x81 substream 0x80) audio 0x8000bd
[23:44:05] scan: decoding previews for title 1
[23:44:05] file is MPEG Program Stream
[23:44:05] Probing 1 unknown stream
[23:44:05]     Probe: Found stream mpegvideo. stream id 0xe0-0x0
[23:44:05] scan: audio 0x8000bd: ac3, rate=48000Hz, bitrate=224000 Unknown (AC3) (2.0 ch) (224 kbps)
[23:44:05] stream: 43 good frames, 0 errors (0%)
[23:44:05] scan: 10 previews, 720x480, 29,970 fps, autocrop = 64/58/0/0, aspect 4:3, PAR 8:9, color profile: 6-6-6
[23:44:05] libhb: scan thread found 1 valid title(s)
[23:44:05] Starting Task: Encoding Pass
[23:44:05] job configuration:
[23:44:05]  * source
[23:44:05]    + /media/starplayer/Sacola1/sotao/History Channel Barbarians & Barbarians 2/BARBARIANSDISC1/VIDEO_TS/VTS_01_1.VOB
[23:44:05]    + title 1, start 00:18:35,00 stop 00:18:55,00
[23:44:05]  * destination
[23:44:05]    + /home/starplayer/Videos/VTS_01_1 (1).mkv
[23:44:05]    + container: Matroska (libavformat)
[23:44:05]  * video track
[23:44:05]    + decoder: mpeg2video 8-bit
[23:44:05]      + bitrate 200 kbps
[23:44:05]    + filters
[23:44:05]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[23:44:05]      + Decomb (mode=47)
[23:44:05]      + Framerate Shaper (mode=1)
[23:44:05]        + frame rate: 29,970 fps -> constant 29,970 fps
[23:44:05]      + Deblock (strength=strong:thresh=20)
[23:44:05]      + Crop and Scale (width=720:height=360:crop-top=64:crop-bottom=56:crop-left=0:crop-right=0)
[23:44:05]        + source: 720 * 480, crop (64/56/0/0): 720 * 360, scale: 720 * 360
[23:44:05]    + Output geometry
[23:44:05]      + storage dimensions: 720 x 360
[23:44:05]      + pixel aspect ratio: 8 : 9
[23:44:05]      + display dimensions: 640 x 360
[23:44:05]    + encoder: H.264 10-bit (libx264)
[23:44:05]      + preset:  medium
[23:44:05]      + profile: auto
[23:44:05]      + level:   auto
[23:44:05]      + quality: 18,00 (RF)
[23:44:05]      + color profile: 6-6-6
[23:44:05]  * audio track 1
[23:44:05]    + decoder: Unknown (AC3) (2.0 ch) (224 kbps) (track 1, id 0x8000bd)
[23:44:05]      + bitrate: 224 kbps, samplerate: 48000 Hz
[23:44:05]    + mixdown: Stereo
[23:44:05]    + encoder: AAC (libavcodec)
[23:44:05]      + bitrate: 160 kbps, samplerate: 48000 Hz
[23:44:05] file is MPEG Program Stream
[23:44:05] Probing 1 unknown stream
[23:44:05]     Probe: Found stream mpegvideo. stream id 0xe0-0x0
[23:44:05] sync: expecting 629 video frames
[23:44:05] encx264: min-keyint: 30, keyint: 300
[23:44:05] encx264: encoding at constant RF 18,000000
x264 [info]: using SAR=8/9
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x264 [info]: profile High 10, level 3.0, 4:2:0, 10-bit
[23:44:05] sync: video time went backwards 33 ms, dropped 1 frames. PTS 1540923
[23:44:14] sync: first pts audio 0x8000bd is 0
[23:44:14] sync: first pts video is 915
[23:44:21] sync: reached audio 0x8000bd pts 1800000, exiting early
[23:44:21] sync: reached video pts 1802715, exiting early
[23:44:21] work: average encoding speed for job is 65,589058 fps
[23:44:21] comb detect: heavy 69 | light 162 | uncombed 369 | total 600
[23:44:21] decomb: deinterlaced 69 | blended 162 | unfiltered 369 | total 600
[23:44:21] vfr: 600 frames output, 0 dropped and 0 duped for CFR/PFR
[23:44:21] vfr: lost time: 0 (0 frames)
[23:44:21] vfr: gained time: 0 (0 frames) (0 not accounted for)
[23:44:21] stream: 367478 good frames, 0 errors (0%)
[23:44:21] ac3-decoder done: 36585 frames, 0 decoder errors
[23:44:21] mpeg2video-decoder done: 35086 frames, 0 decoder errors
[23:44:21] sync: got 600 frames, 629 expected
[23:44:21] sync: framerate min 29,970 fps, max 29,970 fps, avg 29,970 fps
[aac @ 0x7f7b9450a5c0] Ignoring attempt to flush encoder that doesn't support it
[aac @ 0x7f7b9450a5c0] Qavg: 191,727
x264 [info]: frame I:5     Avg QP:29,41  size: 28124
x264 [info]: frame P:152   Avg QP:32,24  size: 17572
x264 [info]: frame B:443   Avg QP:34,64  size:  6990
x264 [info]: consecutive B-frames:  1,0%  1,3%  1,0% 96,7%
x264 [info]: mb I  I16..4:  5,9% 75,7% 18,4%
x264 [info]: mb P  I16..4:  3,0% 19,9%  3,8%  P16..4: 34,9% 24,8% 12,6%  0,0%  0,0%    skip: 1,1%
x264 [info]: mb B  I16..4:  0,5%  2,8%  0,4%  B16..8: 46,7% 13,1%  3,7%  direct:13,5%  skip:19,3%  L0:40,3% L1:40,6% BI:19,1%
x264 [info]: 8x8 transform intra:75,1% inter:74,6%
x264 [info]: coded y,uvDC,uvAC intra: 76,7% 92,0% 57,0% inter: 47,0% 50,2% 3,4%
x264 [info]: i16 v,h,dc,p: 38% 36%  5% 22%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 15% 11%  6% 11% 11% 11% 10% 10%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 25%  8%  5%  9%  9%  8%  7%  5%
x264 [info]: i8c dc,h,v,p: 45% 21% 19% 15%
x264 [info]: Weighted P-Frames: Y:0,0% UV:0,0%
x264 [info]: ref P L0: 64,4% 25,1% 10,5%
x264 [info]: ref B L0: 88,8%  9,0%  2,2%
x264 [info]: ref B L1: 96,4%  3,6%
x264 [info]: kb/s:2360,90
[23:44:21] mux: track 0, 600 frames, 5907927 bytes, 2356,88 kbps, fifo 1024
[23:44:21] mux: track 1, 938 frames, 402927 bytes, 160,74 kbps, fifo 1024
[23:44:22] Finished work at: Mon Jan  3 23:44:22 2022

[23:44:22] libhb: work result = 0

[Skiller]

• It's in the source

• It's not blocking (MPEG compression blocks), it's more like deinterlacing artefacts. Needs special treatment in AviSynth.

[starplayer]

Thank you for your input. It is indeed in the source, albeit not as pronounced. I found it weird that handbrake would intensify these artifacts though.

It's been a few years since I've used AviSynth, I'll have to brush up my skills.

Any tips on modern filters or procedures for it that might apply to this case?

[Skiller]

I'd try NNEDI3 first, like so:

NNEDI3()

Know that this source has been mistreated and we can never recover what was lost because of that, only make what's left look better.

[jagabo]

The guys face doesn't have blocks that a deblocking filter is designed to fix. Fhe the effect of a deblocking filter look at the hand of the guy on the right as it's moving around quickly:


[Attachment 62748 - Click to enlarge]

You want more of an antialiasing filter and noise noise reduction.

[poisondeathray]

Temporal aliasing in the source (deinterlacing artifacts as Skiller mentioned, and compounded by noise issues as jagabo mentioned).

Those artifacts in the source predispose you to even more artifacts when you use lossy encoding

I would use QTGMC in progressive mode , because it smooths over the shimmering and also denoises

Here is gif comparision prior to encoding, illustrating the temporal aliasing artifacts (shimmering)


[Attachment 62749 - Click to enlarge]

qtgmc(preset="very slow", inputtype=1, border=true, sharpness=0.3)

[pcspeak]

Hi all.
Without pushing too hard I did this.
My focus was concentrated on the lady's face in the output.
It looks better to me. I agree with avisynth to take it further.

ffmpeg.exe -i "blocking excerpt original.mkv"  -vf atadenoise,hqdn3d=2:1:2:1 -c:v libx264 -preset slow -crf 18 -c:a copy -threads 0 -y "blocking NEW excerpt original.mkv"

Cheers

[pcspeak]

QTGMC using the latest avisynth+64 to create a deinterlaced, uncompressed avi.
Then the avi put through the ffmpeg script posted above.

https://www.mediafire.com/file/qx5vxlfq35zh8io/blocking+NEW+excerpt+original.mkv/file

Cheers