x264: Tweaking, testing and comparing settings

[Acımasız]

Here is my first x264 guide, it's a short guide about what settings I recommend you tweak to get optimal quality and speed and how to test and compare settings, If you want learn more about x264 I recommend you read Video Encoding Settings for H.264 Excellence (skip to 'Encoder Software - x264'), Awesome-HD - Video Encoding And Ripping X264 Settings and DeathTheSheep's Deblocking Guide.

If you use the default preset…

To get optimal quality I recommend you play around with the AQ mode, AQ strength and PSY-RD. To improve the speed of the default preset (Medium) change some (or all) of these settings --ref, --weightp, --rc-lookahead and --subme. To improve the quality of the default preset (Medium) change some (or all) of these settings --b-adapt, --ref, --rc-lookahead, --me, --direct, --subme and --trellis.

I recommend you start with the default 'slow' preset or these profiles:

Speed: --bframes 16 --ref 4 --rc-lookahead 30 --trellis 2
Quality: --bframes 5 --b-adapt 2 --ref 8 --rc-lookahead 60 --me umh --direct auto --subme 9 --trellis 2

If you use my profiles…

To get optimal quality I recommend you play around with the AQ mode, AQ strength and PSY-RD, if you use U1 v4 also try decreasing the PB Ratio slightly. To improve the speed of U1 v3/4 and U2 v3.5 I recommend you remove some (or all) of these settings --partitionsall, --direct auto, --no-fast-pskip and --b-adapt 2. To improve the speed (quality) of E1/2, mHD v4, U1+ and U2+ try decreasing (increasing) --merange, --bframes and --ref. If you have any memory issues (errors like "memory allocation failed") reduce the --rc-lookahead.

http://freetexthost.com/e0fh1xzabw

Tips:

• Decrease/increase the AQ and PSY-RD strength by 0.05 or 0.1 at a time until you find the optimal amount
• High merange (>32) may decrease quality (the placebo profile only uses 24)
• If you don't need your video to be a specific size use CRF Mode (if you do use 2-pass), for high quality start at 19, for high compression (low bitrate) at <360p start at 22, >360p start at 25 and >720p start at 28, also if you’re using constant ratefactor (--crf) decide the ratefactor first
• If you’re using PSY-Trellis I recommend you also use --trellis 2 and --subme >= 7
• If your using --trellis 2 disabling the DCT Decimation (--no-dct-decimate) may improve quality and decreasing/increasing the deblocking strength/threshold (--deblock 0:0) will probably lower quality
• If your using --b-adapt 2 (optimal) you can use the stats log to find the optimal amount. If your using --b-adapt 1 (fast) use 16, improves quality and doesn't affect the speed
• Increasing the amount of b frames and reference frames may cause compatibility (playback) issues (I’ve added the device presets (from 5i01) at the bottom)
• You can use zones to tweak settings for specific sections of the video

To test settings trim (cut) a few small sections from the video you want to convert (compress) and join them together, then convert the joined sections with the profiles default settings and the setting you want to tweak and compare the results. If the tweak doesn't improve the quality/speed (or you can't see the difference) revert the change you made and move on to the next test, if the tweak does improve the quality (speed) you compare the next tweak you make with this test. I recommend you concentrate on the dark scenes (also the high motion scenes and/or fades if you’re using a low bitrate/high CRF) because these scenes are probable going to be lowest quality or concentrate on the section that’s of the most importance.

If your converting a lot of videos, to save time categorize them into folders first (e.g. Animation 2D (SD) | Animation 2D (HD) | Animation 3D (SD) | Animation 3D (HD)), then trim and join a few small section from several of the videos in each category (folder).

Example 1:

Test 1: Profiles default settings (Slower) at --crf 23 - Bitrate too high and too slow
Test 2: Profiles default settings or --crf increased to 24.5 - Bitrate OK (Test 2)
Test 3: Test 2 or --partitions all removed - Quality OK, still too slow (Test 3)
Test 4: Test 3 or --ref reduced by 2 - Quality OK, still too slow (Test 4)
Test 5: Test 4 or --subme reduced to 8 - Quality and speed OK (Test 5)

Result: --crf 24.5 --b-adapt 2 --ref 6 --rc-lookahead 60 --me umh --direct auto --subme 8 --trellis 2 (Test 5)

Example 2:

Test 1: Profiles default settings or AQ-Mode set to 2 - Profiles default settings
Test 2: AQ strength decreased by 0.1 or AQ strength increased by 0.1 - AQ strength decreased (Test 2)
Test 3: Profiles default settings or AQ strength decreased (Test 2) by 0.1 - Profiles default settings
Test 4: PSY-RDO strength decreased by 0.1 or PSY-RDO increased by 0.1 - PSY-RDO strength decreased (Test 4)
Test 5: Profiles default settings or PSY-RDO strength decreased (Test 4) by 0.1 - Test 4
Test 6: Test 4 or PSY-RDO strength decreased by 0.1 - Test 4
Test 7: Test 4 or PSY-Trellis strength increased by 0.1 - PSY-Trellis strength increased (Test 7)
Test 8: Test 7 or PSY-Trellis strength increased by 0.1 - PSY-Trellis strength increased (Test 8)
Test 9: Test 8 or PSY-Trellis strength increased by 0.1 - Test 8

Result: --aq-mode 1 --aq-strength 1.0 --psy-rd 0.90:0.20 (Test 8)

To save time I recommend you learn the AviSynth and AvsPmod basics. Normally you would have to use up to three programs to trim the video, join the trimmed section together and convert it, with AvsPmod (AviSynth) all you have to do is write a basic script, like this…

DirectShowSource("C:\video.mkv", audio=false)
Trim(1000, 5000) ++ Trim(25000, 29000) ++ Trim(42000, 46000)

Then you can use the 'script encoder' to convert the joined sections (or save the script and use another program to convert it, I recommend MeGUI or TX264).

To compare the results, I recommend you use AvsPmod to compare stills first to see if or where the tweak is making a difference, you can open multiple videos in different tabs or side-by-side and compare the results very easily. Alternatively you can use a program like Thumnail me to capture screenshots, this isn't an accurate method and the results can be misleading (remember to set the output format to PNG (lossless), use one column and to set the width to the same width as the video). After you have compared stills watch both videos and see if the differences are actually noticeable.

Links:

• Encoding for YouTube: How to Get the Best Results
• A&E's Technical Guides to All Things Audio and Video
• https://bytebin.wordpress.com/2011/02/06/x264-presetstunes-and-handbrake/
• http://dev.beandog.org/x264_preset_reference.html
• http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Decoded_picture_buffering
• http://www.scribd.com/doc/34664000/A-Short-Guide-About-Using-Avisynth-MeGUI
• Scintilla's Guide to AVISynth Postprocessing Filters

Device presets:

AVCHD
--profile high --level 4.1 --bluray-compat --open-gop --slices 4 --vbv-bufsize 14000 --vbv-maxrate 14000

Blu-ray
--profile high --level 4.1 --bluray-compat --open-gop --slices 4 --vbv-bufsize 30000 --vbv-maxrate 30000

DivX Plus HD
--profile high --level 4.0 --vbv-bufsize 25000 --vbv-maxrate 20000

DXVA
--profile high --level 4.1 --vbv-bufsize 78125 --vbv-maxrate 62500

iPhone 2G
--profile baseline --level 1.3 --ref 1 --vbv-bufsize 2000 --vbv-maxrate 768

iPhone 3G/3GS, iPod dassic, iPod touch 1/2/3, iPod nano 3/4/5
--profile baseline --level 3.0 --ref 1 --vbv-bufsize 10000 --vbv-maxrate 10000

iPhone 4, iPad 1/2, iPod touch 4/5
--profile main --level 3.1 --vbv-bufsize 14000 --vbv-maxrate 14000

iPhone 4S / 5, iPad 3/4/mini, WDTV
--profile high --level 4.1 --vbv-bufsize 78125 --vbv-maxrate 62500

iPod nano 7
--profile high --level 3.0 --vbv-bufsize 12500 --vbv-maxrate 12500

PS3
--profile high --level 4.1 --vbv-bufsize 31250 --vbv-maxrate 31250

PSP
--profile main --level 3.0 --b-pyramid none --ref 3 --vbv-bufsize 10000 --vbv-maxrate 10000

Xbox 360
--profile high --level 4.1 --bframes 3 --ref 3 --vbv-bufsize 24000 --vbv-maxrate 24000

PSV (Version 1.80)
--profile high --level 4.1 --vbv-bufsize 17500 --vbv-maxrate 17500

[Acımasız]

Just noticed the 'Animated tutorial' in AvsPmod (Help > Animated tutorial) doesn't work, So here's a link to it

http://www.avisynth.nl/users/qwerpoi/Demo.htm

[Acımasız]

Here's the results of a quick test I did between my profiles U1 v3/4 and the default slower preset with tune grain (--profile slower --tune grain) at CRF 19. I should've used 2-pass but it would have took too long and at a high bitrate I don't think the results would've varied that much.

The test video used was Rise of the Planet of the Apes trailer (from Demo World) resized (spline36) from 1920x816 to 1280x544.

Results:

1. Slower/Grain - Avg. Bitrate: 4761.93 kb/s - Avg. Speed: 1.76 fps
2. U1 v3 - Avg. Bitrate: 5329.55 kb/s - Avg. Speed: 3.48 fps
3. U1 v4 - Avg. Bitrate: 5565.95 kb/s - Avg. Speed: 3.35 fps

U1 v3 was 98% faster than Slower/Grain and the bitrate only increased by 12%, U1 v4 was 90% faster and the bitrate increased by 17%.

1. http://files.videohelp.com/u/235547/1.mp4 | SCREENSHOTS
2. http://files.videohelp.com/u/235547/2.mp4 | SCREENSHOTS
3. http://files.videohelp.com/u/235547/3.mp4 | SCREENSHOTS

None of these are great but overall they are very similar visually. The reason why U1 v3 and v4 were a lot faster is because they didn’t use --deblock -2:-2, --ref 8, --subme 9, --me umh or --qcomp 0.8. This doesn’t mean you shouldn’t use these settings (except qcomp*), it just shows you why it's so important to test and compare presets (profiles) and settings first. Also don’t use my profiles based on this test, they are far from perfect and what works for me might not work for you.

Slower/Grain: --crf 19 --deblock -2:-2 --b-adapt 2 --ref 8 --ipratio 1.1 --qcomp 0.8 --rc-lookahead 60 --me umh --direct auto --subme 9 --partitions all --trellis 2 --psy-rd 1.00:0.25 --no-dct-decimate
Universal 1 v3: --level 4.1 --crf 19 --b-adapt 2 --ref 5 --rc-lookahead 90 --aq-mode 2 --aq-strength 1.5 --direct auto --partitions all --trellis 2 --psy-rd 1.00:1.00 --no-dct-decimate --no-fast-pskip
Universal 1 v4: --level 4.1 --crf 19 --b-adapt 2 --ref 5 --qpstep 7 --pbratio 1.66 --no-mbtree --aq-strength 1.5 --direct auto --partitions all --trellis 2 --psy-rd 1.00:1.05 --no-dct-decimate --no-fast-pskip

*http://forum.doom9.org/showthread.php?t=147302
*http://forum.doom9.org/showthread.php?t=147293

[Acımasız]

Here's the results of a quick random test I did between a few of my profiles and the default presets/settings using 2-pass. The video I used was the Fast and Furious 7 trailer resized (spline36) from 1920x804 to 1280x536. To calculate the bitrate I encoded the second 500 frames with my profile and the default preset/settings at my profiles default CRF and used the lowest average as the bitrate (e.g. VF+ at CRF 20.5 average bitrate was 3644Kbps and the Defaults (--preset veryfast) average bitrate was 3025 Kbps, So I used 3025Kbps).

Original (Resized): SCREENSHOTS

mHD v4 (1150Kbps) - Pass 1: 13.97 fps / Pass 2: 4.44 fps / Avg. 9.205 fps | http://files.videohelp.com/u/235547/mHD%20v4.mp4 | SCREENSHOTS
Default (1150Kbps) - Pass 1: 13.95 fps / Pass 2: 4.35 fps / Avg. 9.15 fps | http://files.videohelp.com/u/235547/mHD%20Default.mp4 | SCREENSHOTS

mHD v4: --bframes 4 --b-adapt 2 --ref 4 --rc-lookahead 90 --aq-strength 0.8 --me umh --direct auto --subme 9 --partitions all --trellis 2 --psy-rd 0.85:0 --no-fast-pskip
Default: --bframes 4 --b-adapt 2 --ref 4 --rc-lookahead 90 --me umh --direct auto --subme 9 --partitions all --trellis 2 --no-fast-pskip

VF+ (3025Kbps) - Pass 1: 17.93 fps / Pass 2: 16.39 fps / Avg. 17.16 fps | http://files.videohelp.com/u/235547/VF_.mp4 | SCREENSHOTS
Default (3025Kbps) - Pass 1: 18.11 fps / Pass 2: 16.17 fps / Avg. 17.14 fps | http://files.videohelp.com/u/235547/VF.mp4 | SCREENSHOTS

VF+: --preset veryfast --bframes 16 --deadzone-inter 15 --deadzone-intra 5 --no-dct-decimate
Default: --preset veryfast

U2 v3.5 (2043Kbps) - Pass 1: 14.50 fps / Pass 2: 5.08 fps / Avg. 9.79 fps | http://files.videohelp.com/u/235547/U2%20v3.5.mp4 | SCREENSHOTS
Slow (2043Kbps) - Pass 1: 14.62 fps / Pass 2: 4.27 fps / Avg. 9.445 fps | http://files.videohelp.com/u/235547/Slow.mp4 | SCREENSHOTS

U2 v3.5: --level 4.1 --b-adapt 2 --ref 5 --rc-lookahead 90 --aq-mode 2 --direct auto --partitions all --trellis 2 --no-dct-decimate --no-fast-pskip
Slow: --preset slow

U2 v3.5 + Psy-Trellis (2135Kbps) - Pass 1: 14.96 fps / Pass 2: 5.21 fps / Avg. 10.085 fps | http://files.videohelp.com/u/235547/U2%20v3.5%20_%20Psy-Trellis.mp4 | SCREENSHOTS
Slow + Psy-Trellis (2135Kbps) - Pass 1: 15.12 fps / Pass 2: 4.40 fps / Avg. 9.76 fps | http://files.videohelp.com/u/235547/Slow%20_%20Psy-Trellis.mp4 | SCREENSHOTS

U2 v3.5: --level 4.1 --b-adapt 2 --ref 5 --rc-lookahead 90 --aq-mode 2 --direct auto --partitions all --trellis 2 --psy-rd 1.00:0.15 --no-dct-decimate --no-fast-pskip
Slow: --preset slow --psy-rd 1.00:0.15

[Acımasız]

Thought id also test 10 Bit and AQ Mode 3 (x264 core 144 r2525kMod - http://komisar.gin.by/)

Original (Resized): SCREENSHOTS

U2 v3.5 (2043Kbps) - Pass 1: 14.90 fps / Pass 2: 5.24 fps / Avg. 10.07 fps | http://files.videohelp.com/u/235547/U2%20v3.mp4 | SCREENSHOTS
U2 v3.5 + AQ3 (2043Kbps) - Pass 1: 14.67 fps / Pass 2: 5.08 fps / Avg. 9.875 fps | http://files.videohelp.com/u/235547/U2%20v3%20AQ3.mp4 | SCREENSHOTS
U2 v3.5 (10 Bit / 2043Kbps) - Pass 1: 11.04 fps / Pass 2: 4.27 fps / Avg. 7.655 fps | http://files.videohelp.com/u/235547/U2%20v3%2010b.mp4 | SCREENSHOTS
U2 v3.5 + AQ3 (10 Bit / 2043Kbps) - Pass 1: 10.97 fps / Pass 2: 3.62 fps / Avg. 7.295 fps | http://files.videohelp.com/u/235547/U2%20v3.5%2010b%20_%20AQ3.mp4 | SCREENSHOTS

U2 v3.5: --level 4.1 --b-adapt 2 --ref 5 --rc-lookahead 90 --aq-mode 2 (3) --direct auto --partitions all --trellis 2 --no-dct-decimate --no-fast-pskip

[Acımasız]

Here are my profiles for Handbrake 0.10.0/0.10.1. If you spot any mistakes or have any issues send me a PM.

http://freetexthost.com/e0fh1xzabw

Extract user_presets.xml into the Handbrake install/config directory...

Windows 7/8?: Win+R (Start > Run) > %userprofile%\AppData\Roaming\HandBrake or copy and paste this into the explorer address bar; C:\Users\%username%\AppData\Roaming\HandBrake

Windows Vista: Win+R (Start > Run) > %userprofile%\AppData\HandBrake or copy and paste this into the explorer address bar; C:\Users\%username%\AppData\HandBrake

If you have your own presets and want to keep them; Extract user_preset.xml to your desktop and open it in notepad, copy all the text except the first two lines ( , ? XML ... and < ArrayOf ... ) and close notepad, then go to the Handbrake install/config directory and open user_preset.xml in notepad, go to the end, delete the last line ( < / ArrayOfPreset > ), paste the text you copied from the first user_preset.xml and save it (Ctrl+S).

[Acımasız]

Updated my profiles

http://freetexthost.com/e0fh1xzabw

[8day]

I recommend you to add two pages from x265 readthedocs.com site. I haven't seen such a good explanation for some of x264 options anywhere else (as you probably know, x264 & x265 share not only same settings, but to some extent even the code base). I really liked part about psy-rd, which is one of the most useful options in x264:

• http://x265.readthedocs.org/en/default/cli.html
• http://x265.readthedocs.org/en/default/presets.html

Also AviDemux guide (http://www.avidemux.org/admWiki/doku.php?id=tutorial:h.264) may provide a good alternative POV on some things. Another useful alternative guide, on top of which more recent ones may have been based, is DeathTheSheep's AVC VfW Guide (https://sites.google.com/site/gabextreme/x264vfwunited). Incidentally that VfW guide includes deblocking guide mentioned in the first post.

P.S. Thank you for the link to the Lighterra guide!