Video encoding help

[Aurora001]

I'm new to the world of encoding and H.264, so please go easy on me.
I've decided to encode my blu-ray library into MKV containers for portability purposes, however, I'm getting far too many artifacts on the encode as you can see below:

Original



Encode (x264 - Handbrake & MeGUI - both 10 bit & 8 bit):



The blue pattern a the top is totally jagged with the encode. Any tips to fix this problem? I've tried both Handbrake and MeGUI, 8 bit x264 and 10 bit x264, but cannot seem to get a good encode.

By comparison, here's an encode that I found online of the above scene by another group:



What am I doing wrong? How do I achieve a similar encode quality (with no artifacts)? Both encodes are at CRF = 18.0

Further information:

Legend of Korra: Book 1 - Air (Blu-ray)

My encode's media information

Format                                   : Matroska
Format version                           : Version 2
File size                                : 105 MiB
Duration                                 : 1mn 0s
Overall bit rate                         : 14.6 Mbps
Writing application                      : HandBrake 20160508174001-d61a9be-master 2016051101
Writing library                          : Lavf56.1.0 / Lavf56.1.0

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High 10@L4
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 1mn 0s
Bit rate                                 : 14.3 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.288
Stream size                              : 103 MiB (98%)
Writing library                          : x264 core 144
Encoding settings                        : cabac=1 / ref=4 / deblock=1:-3:-3 / analyse=0x3:0x133 / me=tesa / subme=11 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=1 / keyint=240 / keyint_min=24 / scenecut=40 / intra_refresh=0 / rc_lookahead=240 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.70 / qpmin=0 / qpmax=81 / qpstep=4 / ip_ratio=1.40 / aq=2:1.00
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Online group's media information

General
Format                                   : Matroska
Format version                           : Version 4 / Version 2
File size                                : 726 MiB
Duration                                 : 24mn 4s
Overall bit rate                         : 4 216 Kbps
Encoded date                             : UTC 2013-07-10 17:12:11
Writing application                      : mkvmerge v6.3.0 ('You can't stop me!') built on Jun 28 2013 20:09:41
Writing library                          : libebml v1.3.0 + libmatroska v1.4.0

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 24mn 4s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Writing library                          : x264 core 133 r2334 a3ac64b
Encoding settings                        : cabac=1 / ref=4 / deblock=1:-3:-3 / analyse=0x3:0x133 / me=tesa / subme=10 / psy=0 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=0 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=9 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=2 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc=crf / mbtree=0 / crf=18.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / pb_ratio=1.30 / aq=2:0.60
Language                                 : English
Default                                  : Yes
Forced                                   : No

Audio #1
ID                                       : 2
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : A_AC3
Duration                                 : 24mn 4s
Bit rate mode                            : Constant
Bit rate                                 : 640 Kbps
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 KHz
Frame rate                               : 31.250 fps (1536 spf)
Compression mode                         : Lossy
Stream size                              : 110 MiB (15%)
Title                                    : DD5.1
Language                                 : English
Default                                  : Yes
Forced                                   : No

[jagabo]

Your problem has nothing to do with h.264 encoding. It's a deinterlacing issue. Try using decomb instead of deinterlace in HandBrake.

[Aurora001]

Your problem has nothing to do with h.264 encoding. It's a deinterlacing issue. Try using decomb instead of deinterlace in HandBrake.

Thanks. How do I use decomb, which settings do I input? This is my first ever encode. Here's a screenshot of Handbrake:



Edit:

I'm still getting the weird distortions at the top with Decomb (Default) & Interlace Detection: Default.



Any help please?

[hello_hello]

It might pay to upload a small sample of the problem video. I'm not certain as to when/if you're resizing either, given the screenshots aren't 1080p. It probably wouldn't hurt to keep them at their original size.

If the source video is 23.976fps or 24fps progressive, try disabling Handbrake's Decomb and De-interlacing filters completely.

I assume you have the Bluray ripped to your hard drive? You can use programs such a MKVToolNixGUI or TSMuxer to remux the source file (just the video would be fine). Both have a splitting function. You only need to upload a small section from the original video where you're having problems.

[Bernix]

Hi,
have you any good reason to convert it to 10bit Bit depth instead of 8bit? And you can try to lower AQ strenght to 0.5-0.6. .
And 16 bframes is too much. I usually used max 5.

Bernix

[jagabo]

We'll have to see a sample of your source to give further advice.

[Aurora001]

Thank you for your feedback everyone.

Here's a short sample of the original video - I re-muxed it into an MKV and split a short ~10 sec relevant portion. Link (Download through your browser)

I cannot get my encode to look clean as the original - always get jagged/distortions as indicated in the OP. Any help is appreciated.

P.S: I tried both 8 bit and 10 bit encodes with no luck. I tried MeGUI with no luck. Tried different decomb settings, etc.

[manono]

Why are you doing anything to it? It's already progressive. It doesn't need 'decombing', 'detelecining', 'deinterlacing' or anything else.

[Bernix]

Just try, I have very slow computer

[Aurora001]

Why are you doing anything to it? It's already progressive. It doesn't need 'decombing', 'detelecining', 'deinterlacing' or anything else.

I did not use any filters but ended up with this:
http://screenshotcomparison.com/comparison/172870

Any ideas?

Just try, I have very slow computer

Wow. Your version is a lot cleaner (and dare I say almost identical to the source). Which settings did you use or how did you fix the issue?

[Bernix]

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5.1
Format settings, CABAC : Yes
Format settings, ReFrames : 16 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 9s 301ms
Bit rate : 25.5 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (23976/1000) fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.512
Stream size : 28.2 MiB (98%)
Writing library : x264 core 148 r2665kMod a01e339
Encoding settings : opencl=1 / cabac=1 / ref=16 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=9 / psy=0 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0 / threads=2 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / fgo=0 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc=cqp / mbtree=0 / qp=18 / ip_ratio=1.40 / pb_ratio=1.30 / aq=0
Language : English
Default : Yes
Forced : No

[KarMa]

Why are you doing anything to it? It's already progressive. It doesn't need 'decombing', 'detelecining', 'deinterlacing' or anything else.

I did not use any filters but ended up with this:
http://screenshotcomparison.com/comparison/172870

Any ideas?

Deinterlace drop down menu was set to off?

[hello_hello]

Here's your sample opened in MeGUI's preview via Avisynth (open the images in new tabs or save them to view full size):



Here it is again opened in MeGUI's preview after MeGUI analysed it and incorrectly decided it was interlaced. Nothing encoded. Just decoded and with de-interlacing enabled:



If you're using MeGUI's script creator make sure there's no de-interlacing enabled.
If you're using the OneClick encoder, open it, select "show advanced settings" and make sure "automatic de-interlacing" is unchecked, due to the "getting it wrong" problem (I don't know for sure if that'd be the cause as I never use the OneClick encoder myself).
If you happen to be using DirectShowSource for opening the video, probably better not to, but if you are, make sure your DirectShow decoder isn't doing any de-interlacing. That's out of MeGUI's control.

I don't know what Handbrake is doing but logically there must still be some sort of de-interlacing enabled.

Edit: I opened the sample with MeGUI's File Indexer and indexed it with L-Smash.

[jagabo]

Yes, the source is progressive so no deinterlacing or decombing should be performed. Disable all such filters in Handbrake. Try the x264 slow or slower preset at about CRF 18.

[manono]

I did not use any filters.

I don't believe you. Not that you're lying, but that you filtered it accidentally. Check the MeGUI AviSynth script it used.

[Aurora001]

Thank you everyone!

Managed to resolve the issue. Both Handbrake/MeGUI had deinterlacing/decombing enabled by default. Once I turned it off, the encode appears acceptable now:

http://screenshotcomparison.com/comparison/172872

Settings used:

x264 10 bit
CRF = 18.0
Constant Frame Rate (same as source)



Encode (57% size reduction)

[Bernix]

I think it is not good idea to convert it to 10 bit depth.
But it is only my opinion.
You can try MakeMkv maybe you will find it useful.

[jagabo]

Yes, 10 bit and 9 b-frames will kill compatibility with most hardware, like blu-ray players.

[hello_hello]

Yes, you definitely should be aware if you use 10 bit encoding you're pretty much going to be limited to PC playback. There's almost no hardware player support.

10 bit decoding will be more likely to become mainstream as h265 does, as 10 bit has been part of the HEVC standard from the beginning. There's no guarantee it'll ever become mainstream for AVC. I suspect it won't.

[Aurora001]

I think it is not good idea to convert it to 10 bit depth.
But it is only my opinion.
You can try MakeMkv maybe you will find it useful.

Yes, 10 bit and 9 b-frames will kill compatibility with most hardware, like blu-ray players.

Yes, you definitely should be aware if you use 10 bit encoding you're pretty much going to be limited to PC playback. There's almost no hardware player support.

10 bit decoding will be more likely to become mainstream as h265 does, as 10 bit has been part of the HEVC standard from the beginning. There's no guarantee it'll ever become mainstream for AVC. I suspect it won't.

I read that 10bit has better compression efficiency and mitigates banding issues.
Compatibility issues are a valid concern. I'll be primary viewing/using these encodes on an iPad Air with VLC. It's more portable than a PC.

Whether or not it can playback 1080p x264 10bit video with 24bit 5.1 FLAC audio, I'll find out tomorrow.

[hello_hello]

As a test I encoded the sample using the 8 bit encoder, Tune Film, Preset Very Slow and High Profile level 4.1.
The file size was exactly the same as the your sample 10 bit encode and they're very, very similar in quality. Comparing them frame by frame I could see small differences in the way some frames were encoded but neither was "better". Sometimes the frame from the 8 bit encode matched the source frame more closely, sometimes it was the 10 bit frame, but that's always going to happen when you use different settings whether the encoder is 8 bit or 10 bit.

Personally, I'd keep "anything can play it" as a higher priority the banding (which is an issue) but that's just me.

These days, I stick gradfin3() at the end of most scripts (from the dither package) and always when using noise removal, and colour banding isn't much of an issue.

[Aurora001]

As a test I encoded the sample using the 8 bit encoder, Tune Film, Preset Very Slow and High Profile level 4.1.
The file size was exactly the same as the your sample 10 bit encode and they're very, very similar in quality. Comparing them frame by frame I could see small differences in the way some frames were encoded but neither was "better". Sometimes the frame from the 8 bit encode matched the source frame more closely, sometimes it was the 10 bit frame, but that's always going to happen when you use different settings whether the encoder is 8 bit or 10 bit.

Personally, I'd keep "anything can play it" as a higher priority the banding (which is an issue) but that's just me.

These days, I stick gradfin3() at the end of most scripts (from the dither package) and always when using noise removal, and colour banding isn't much of an issue.

Thanks for sharing your experience. Appreciate it.

I'm re-encoding via 8bit x264 now. My 10bit encode crashed overnight (ran out of memory). 10bit and rc-lookahead = 240 may not have been a good idea with only 8 GB of memory. The program was using 7 GB of RAM before it crashed.

Handbrake's encoding fine now, with 1123 MB memory usage.