HD Video Compression. Best Quality @ 1/3 Size. Help

[ampreston85]

Alright, so I have been at this for quite some time and am pretty familiar with HD video and audio. I have been using Handbrake for a couple of years now. Now I am dealing directly with either A. raw .M2TS files (where the bluray doesn't have the feature film split into multiple), and B. more commonly, raw/uncompressed .MKV files. Ultimately I'm trying to conserve GB's where ever possible with MINIMAL quality loss. I have several bluray movies which are 25-35 GB's in size, and want them to be compressed down to around 1/3 the size. I have fiddled around with Handbrakes 'Advanced' Settings. I'm not too picky with the file size being an exact size, knowing that larger movies are gonna be around 10-15 GB and the shorter movies will be around 8-10 GB in size. I just don't know exactly what combination of these 'Advanced' Settings to change around. The media player I am watching these files on, has no problem playing through any profile/level I have thrown at it this far (Example: plays Transformers 3 which is uncompressed at 45 GB & 40+ MB bitrate, as well as The Bourne Ultimatum which is rated High@L5.1 w/ 8 Reference Frames) both of which exceed the "maximum" ability of my media player. Bottom line, I would like some recommendations as to what settings to tweek, and what to tweek them to. I prefer to leave audio uncompressed as I only put in the DTS 1510 KPBS stream and remove all others. I appreciate your responses

[ampreston85]

These are my current settings. My RF is set to 15, though I may slightly increase that as my most recent compression was still 65% of the original file size.

[jagabo]

Most of the settings don't make much difference past those of the x264 "slow" preset. List of x264 presets (may not be fully up to date since they occaionsally change):

ultrafast: cabac=0 / ref=1  / deblock=0:0:0 / analyse=0:0       / me=dia  / subme=0  / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=0 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0  / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0                                                                           / weightp=0 / keyint=250 / keyint_min=23 / scenecut=0  / intra_refresh=0                   / rc=crf / mbtree=0 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=0
superfast: cabac=1 / ref=1  / deblock=1:0:0 / analyse=0x3:0x3   / me=dia  / subme=1  / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0  / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3  / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=1 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0                   / rc=crf / mbtree=0 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / pb_ratio=1.30 / aq=1:1.00
veryfast:  cabac=1 / ref=1  / deblock=1:0:0 / analyse=0x3:0x113 / me=hex  / subme=2  / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0  / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3  / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=1 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=10 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=1:1.00
faster:    cabac=1 / ref=2  / deblock=1:0:0 / analyse=0x3:0x113 / me=hex  / subme=4  / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0  / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3  / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=1 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=20 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=1:1.00
fast:      cabac=1 / ref=2  / deblock=1:0:0 / analyse=0x3:0x113 / me=hex  / subme=6  / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3  / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=1 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=30 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=1:1.00
medium:    cabac=1 / ref=3  / deblock=1:0:0 / analyse=0x3:0x113 / me=hex  / subme=7  / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3  / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=1:1.00
slow:      cabac=1 / ref=5  / deblock=1:0:0 / analyse=0x3:0x113 / me=umh  / subme=8  / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3  / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=1:1.00
veryslow:  cabac=1 / ref=16 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh  / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=8  / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=1:1.00
placebo:   cabac=1 / ref=16 / deblock=1:0:0 / analyse=0x3:0x133 / me=tesa / subme=11 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40                 / aq=1:1.00

Stepping me_range up (as you have) will help a bit with HD video. Bumping up aq-strength will help retain grain in dark scenes -- reducing posterization artifacts. When using CRF encoding the bitrate will shoot up though.

[poisondeathray]

Filtering (e.g. degrain, denoise) it will reduce the size at a given CRF and reduce the bitrate requirements . Very important if you're shooting for 1/3 the size.

[ampreston85]

Alright, so I have tweaked the settings a bit, as the previous settings I've posted made the encode time 45 hours LOL (Running Intel E8500 Dual Core 3.17ghz CPU w/ 2GB DDR3 1600mhz RAM). Here are my modified settings. Encode time is approx 18 hours (anything less than 1 day is good with me). Quality was virtually identical to original HD source. Only remaining problem is output file size. Still a bit on the large size. I took the movie, The Perks of Being a Wall Flower @ 28.8 GB. on disk, and after encode, it was still 17.5 GB. I'm hoping I can just adjust the RF up from 20 to like... 25 or so, and this should bring down target size to around what I'm looking for (~ 12 GB)? Any and all thoughts/ideas are welcome. Thank you again for your help!

[jagabo]

Raising AQ Strength is probably costing you a lot of bitrate. But it helps retain noise/grain, resulting in less posterization, especially in dark areas. Subme 8 is probably only getting less than 1 percent better compression than subme 6, but costing a lot in encoding time.

[ampreston85]

Meh I'm okay with the longer time given that I start the process in the a.m and don't get back to my PC usually until the next morning. I'm picky about the quality and I'm hoping that I won't get to that point where I will have to give up significant quality loss to obtain the desired output file size. I have Prometheus going right now. Only changed the CRF to 25 and we shall see what that leaves me with. I was thinking of going down to 4 B Frames... Or maybe turning Trellis off completely?

[jagabo]

Raising the CRF value is going make much more difference in size (smaller) and quality (lower) than fiddling with any of the other settings (unless you are down in the superfast or ultrafast preset range).