How to reduce blocking/artifacts in anime encode?

[Pepsi_Doggo]

Hi, I just register to the forum, I've been trying for days to encode the first episode of Flip Flappers but I keep getting this weird artifacts in the dark scenes so I downloaded a release of a anime encoding group( Good Job Media), in their encode of the first episode they almost got rid of all these artifacts, I wanted to know which x264 setting I have to tweak to get rid of the artifacts.
In the BDMV these artifacts are also present but are less noticeable.
BDMV:

Full size:https://forum.videohelp.com/images/imgfiles/Z4grRoi.png

My encode:

Full size:https://forum.videohelp.com/images/imgfiles/uUjpNvC.png

My settings:8-bit/Chroma subsampling: 4:2:0

cabac=1 / ref=16 / deblock=1:-1:-2 / analyse=0x3:0x133 / me=tesa / subme=11 / psy=1 / psy_rd=0.70:0.00 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=24 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=15.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=300000 / vbv_bufsize=300000 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00

Good Job Media encode:

Full size:https://forum.videohelp.com/images/imgfiles/Hl1Jx3G.png

Good Job Media settings:10-bit/Chroma subsampling: 4:4:4

cabac=1 / ref=16 / deblock=1:-1:-2 / analyse=0x3:0x133 / me=tesa / subme=10 / psy=1 / fade_compensate=0.70 / psy_rd=0.70:0.00 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-1 / threads=9 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=0 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / fgo=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=80 / rc=crf / mbtree=1 / crf=16.5000 / qcomp=0.68 / qpmin=0:0:0 / qpmax=81:81:81 / qpstep=4 / ip_ratio=1.40 / aq=3:0.85 / aq-sensitivity=10.00 / aq-factor=1.00:1.00:1.00 / aq2=0 / aq3=0

I'm using Handbrake 1.0.3 64-bit

[poisondeathray]

They are using a different x264 build with the fade compensate patch, and aq3 patch. But you can't drop in and replace the binary with handbrake, you'd have to use something other than handbrake like a custom x264cli directly

10bit encoding also helps alot with banding/blocking along gradients. You're using 8bit. Maybe one of the handbrake nightly builds offers 10bit, not sure. You're probably better of using something else if it doesn't