x265 HEVC Encoder

[LigH.de]

Some of these options are exposed for technically advanced testing and may be revoked later, according to discussions I "overheard"...

[x265]

Since the x265 folks seem to ignore my questions at all here are my thoughts about the options:

--cu-stats, --no-cu-stats

Records statistics on how each CU was coded (split depths and other mode decisions) and reports those statistics at the end of the encode. Default disabled

Is there any gain in using this for a non-developer?

--hrd, --no-hrd

Enable the signalling of HRD parameters to the decoder. The HRD parameters are carried by the Buffering Period SEI messages and Picture Timing SEI messages providing timing information to the decoder. Default disabled

okay, is there any use for this at the current time?

--ipratio <float>

QP ratio factor between I and P slices. This ratio is used in all of the rate control modes. Some --tune options may change the default value. It is not typically manually specified. Default 1.4

QP ratio factor between P and B slices. This ratio is used in all of the rate control modes. Some --tune options may change the default value. It is not typically manually specified. Default 1.3

okay (hopefully there will be a documentation about the --tune option in the future which shows what they do)

--lambda-file <filename>

Specify a text file containing values for x265_lambda_tab and x265_lambda2_tab. Each table requires MAX_MAX_QP+1 (70) float values.

The text file syntax is simple. Comma is considered to be white-space. All white-space is ignored. Lines must be less than 2k bytes in length. Content following hash (#) characters are ignored. The values read from the file are logged at --log-level debug.

Note that the lambda tables are process-global and so the new values affect all encoders running in the same process.

Lambda values affect encoder mode decisions, the lower the lambda the more bits it will try to spend on signaling information (motion vectors and splits) and less on residual. This feature is intended for experimentation.

okay, so no way a normal user will/can use this.

Professionals (movie studios, etc.) require HRD conformant HEVC bitstreams. This information is useful to video processing and analytical tools downstream of the encoder. See Annex C of the HEVC specifications for more info on HRD (Hypothetical Reference Decoder).

The rest of the new options are for encoder research and development. They enable us to run tests more easily (without creating separate builds of x265), and to capture more information as we run tests.

Tom

[Selur]

would be nice if the list of open questions I posted in previous post could also be answered,...

[vhelp]

@ x265 team, please consider adding this feature in: recording the x265 encoding parameters to the h265 bitstream.

i commented about adding x265 encoding parameters in the thread below. i know i've asked about this feature before but i think it high time to incorporate this feature into the x265.exe encoder soon. but for now, we can use ffmpeg to add them to the mp4 container. i've tried adding to mkv but the latest version of ffmpeg does not work properly, and mvktoolnix can write them but it seems to complex for simple command line or scripting. it was easier for me to incorporate this into my gui front-end, but the ffmpeg method should work for most people that want to record their parameters for later encoding analyzing and review purposes.

https://forum.videohelp.com/threads/365809-adding-x265-encoding-parameters-via-metadata?p=2333392

[x265]

@ x265 team, please consider adding this feature in: recording the x265 encoding parameters to the h265 bitstream.

i commented about adding x265 encoding parameters in the thread below. i know i've asked about this feature before but i think it high time to incorporate this feature into the x265.exe encoder soon. but for now, we can use ffmpeg to add them to the mp4 container. i've tried adding to mkv but the latest version of ffmpeg does not work properly, and mvktoolnix can write them but it seems to complex for simple command line or scripting. it was easier for me to incorporate this into my gui front-end, but the ffmpeg method should work for most people that want to record their parameters for later encoding analyzing and review purposes.

https://forum.videohelp.com/threads/365809-adding-x265-encoding-parameters-via-metadata?p=2333392

Someone on Doom9 requested this earlier today (you?), and we discussed it internally and externally, and we agreed that although it was already on our to-do list, we should make it a higher priority. Steve was kind enough to take care of it today, and a patch is queued for testing now.

[vhelp]

no, that wasn't me over there. sorry for the confusion.

that is good news, thank you guys, steve, and everyone else envolved in the project

now all that is next then, is to be able to see that info via mediainfo and ffmpeg. i don't know of any other tool that shows that info, unless x265.exe will include something in the output report, a parameter switch, for instance, if you run x265 -info video.hm10. or else, maybe an updated ffmpeg will follow. either way, great news that this feature will be available now

[LigH.de]

Please note that there may be a difference between CLI parameters and API parameters. Like in x264, there are "simple" and "complex" parameter; a usual example for a "complex" parameter will be "preset", which will be separated into a set of simple parameters before they are sent to the encoder. Including default values. The metadata in x264 AVC output will contain only API parameters, which may even not be equal to simple CLI parameters in all cases. You will never find "preset" or "tune" in the metadata. So I believe x265 may have to do it the same way, to contain useful information even if it was used as DLL or linked-in library in an own application (similar to ffmpeg), not as the currently tested CLI encoder (x265.exe).

[El Heggunte]

vhelp@forum.doom9.org == http://forum.doom9.org/member.php?u=26605

Therefore, nope, it wasn't him

EDIT: And below is the conspiracy's log

[Selur]

did anyone test the new 2pass encoding support in x265?

[LigH.de]

In which build was that exposed? Not yet in v1.2+239, according to the help text. Building v1.2+277 now...
__

No, v1.2+277 does not yet expose a "--pass" parameter; and I don't spot any other parameter related to 2-pass encoding.

[Selur]

two-pass encoding is now functional on the current tip; the last relevant commit was <<cset b85dbec30cc5>>

source: https://bitbucket.org/multicoreware/x265/issue/38/2pass-encoding-support

[LigH.de]

But there is no command line parameter handling yet to use it in the x265 CLI encoder.

[Selur]

Ohhh,... seems like functional only referred to some internals
(added a comment to the ticket)

[El Heggunte]

0_o

# HG changeset patch
# User Aarthi Thirumalai
# Date 1406218558 -19800
#      Thu Jul 24 21:45:58 2014 +0530
# Node ID 5d56988ebdcfdd56ef9960f4b6cf5d352e1c340c
# Parent  47407360120a1d92478bd67c082ddc7df8ea1932
rc: add cli option for 2 pass rate control

diff -r 47407360120a -r 5d56988ebdcf source/common/param.cpp
--- a/source/common/param.cpp	Wed Jul 23 23:57:59 2014 -0500
+++ b/source/common/param.cpp	Thu Jul 24 21:45:58 2014 +0530
@@ -755,6 +755,16 @@
                          &p->vui.defDispWinBottomOffset) != 4;
     }
     OPT("nr") p->noiseReduction = atoi(value);
+    OPT("pass")
+    {
+        int pass = Clip3(0, 3, atoi(value));
+        p->rc.bStatWrite = pass & 1;
+        p->rc.bStatRead = pass & 2;
+    }
+    OPT("stats")
+    {
+        p->rc.statFileName = strdup(value);
+    }
     else
         return X265_PARAM_BAD_NAME;
 #undef OPT
diff -r 47407360120a -r 5d56988ebdcf source/x265.cpp
--- a/source/x265.cpp	Wed Jul 23 23:57:59 2014 -0500
+++ b/source/x265.cpp	Thu Jul 24 21:45:58 2014 +0530
@@ -194,6 +194,8 @@
     { "b-intra",              no_argument, NULL, 0 },
     { "no-b-intra",           no_argument, NULL, 0 },
     { "nr",             required_argument, NULL, 0 },
+    { "stats",          required_argument, NULL, 0 },
+    { "pass",           required_argument, NULL, 0 },
     { 0, 0, 0, 0 }
 };
 
@@ -412,6 +414,11 @@
     H0("   --cbqpoffs <integer>          Chroma Cb QP Offset. Default 
%d\n", param->cbQpOffset);
     H0("   --crqpoffs <integer>          Chroma Cr QP Offset. Default 
%d\n", param->crQpOffset);
     H0("   --[no-]hrd                    Enable HRD parameters 
signalling. Default %s\n", OPT(param->bEmitHRDSEI));
+    H0("   --stats                       FileName for stats file in 
multipass pass rate control. Default %s\n", 
OPT(param->rc.statFileName));
+    H0("   --pass                        Multi pass rate control.\n"
+       "                                   - 1 : First pass , cretes 
stats file\n"
+       "                                   - 2 : Last pass, does not 
overwrite stats file\n"
+       "                                   - 3 : Nth pass, overwrites 
stats file\n");
     H0("   --rd <0..6>                   Level of RD in mode decision 
0:least....6:full RDO. Default %d\n", param->rdLevel);
     H0("   --psy-rd <0..2.0>             Strength of psycho-visual 
optimization. Requires slow preset or below. Default %f\n", 
param->psyRd);
     H0("   --[no-]signhide               Hide sign bit of one coeff per 
TU (rdo). Default %s\n", OPT(param->bEnableSignHiding));

[Selur]

nice

[x265]

2 pass is enabled in the Development tip, but it's not fully debugged yet. Feel free to try it, and let us know your feedback. At the moment, it doesn't play nice with VBV on the 2nd pass (which aborts instantly).

[Selur]

"G:\Hybrid\x265-16bit.exe" --input - --input-depth 10 --input-res 720x364 --fps 23.976 --frames 203664 --pass 1 --bitrate 998 --colormatrix bt470bg --stats "H:\Temp\test_08_41_35_7910_08.stats" --output NUL
x265 [info]: HEVC encoder version 1.2+305-66ed81577483
x265 [info]: build info [Windows][GCC 4.9.1][64 bit] 16bpp
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x265 [info]: WPP streams / pool / frames : 6 / 8 / 3
x265 [info]: Main10 profile, Level-3 (Main tier)
x265 [info]: Internal bit depth : 10
x265 [info]: CU size : 64
x265 [info]: Max RQT depth inter / intra : 1 / 1
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut : 23 / 250 / 40
x265 [info]: Lookahead / bframes / badapt : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb / refs: 1 / 1 / 0 / 3
x265 [info]: Rate Control / AQ-Strength / CUTree : ABR-998 kbps / 1.0 / 1
x265 [info]: tools: rd=3 lft sao-lcu signhide stats-write
encoded 203664 frames in 5159.26s (39.48 fps), 995.87 kb/s
x265 [info]: frame I: 2496, Avg QP:21.00 kb/s: 6736.31
x265 [info]: frame P: 56779, Avg QP:22.69 kb/s: 2179.62
x265 [info]: frame B: 144389, Avg QP:27.74 kb/s: 431.15
x265 [info]: global : 203664, Avg QP:26.25 kb/s: 995.87
x265 [info]: Weighted P-Frames: Y:5.1% UV:4.1%
x265 [info]: consecutive B-frames: 22.6% 6.3% 13.0% 20.9% 37.2%
finished after 01:25:59.358

"G:\Hybrid\x265-16bit.exe" --input - --input-depth 10 --input-res 720x364 --fps 23.976 --frames 203664 --pass 2 --bitrate 998 --colormatrix bt470bg --stats "H:\Temp\test_08_41_35_7910_08.stats" --output "H:\Temp\test_08_41_35_7910_09.265"
x265 [info]: HEVC encoder version 1.2+305-66ed81577483
x265 [info]: build info [Windows][GCC 4.9.1][64 bit] 16bpp
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x265 [info]: WPP streams / pool / frames : 6 / 8 / 3
x265 [info]: Main10 profile, Level-3 (Main tier)
x265 [info]: Internal bit depth : 10
x265 [info]: CU size : 64
x265 [info]: Max RQT depth inter / intra : 1 / 1
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut : 23 / 250 / 40
x265 [info]: Lookahead / bframes / badapt : 0 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb / refs: 1 / 1 / 0 / 3
x265 [info]: Rate Control / AQ-Strength / CUTree : ABR-998 kbps / 1.0 / 1
x265 [info]: tools: rd=3 lft sao-lcu signhide stats-read
encoded 203664 frames in 4864.89s (41.86 fps), 1004.99 kb/s
x265 [info]: frame I: 2496, Avg QP:19.66 kb/s: 7570.49
x265 [info]: frame P: 56779, Avg QP:22.88 kb/s: 2132.16
x265 [info]: frame B: 144389, Avg QP:28.01 kb/s: 448.26
x265 [info]: global : 203664, Avg QP:26.48 kb/s: 1004.99
x265 [info]: Weighted P-Frames: Y:2.7% UV:2.1%
x265 [info]: consecutive B-frames: 22.6% 6.3% 13.0% 20.9% 37.2%
finished after 01:21:10.972
Created H:\Temp\test_08_41_35_7910_09.265 (1018.48 MB)

-> 1st pass is a bit too low, but near the target bit rate.
Problem is 2nd pass overshoot the target bit rate.
Aimed for 998 kbps, got 1004.99 kb/s.

Is there an option to make x265 stick more to the desired bit rate and worst case undershoot, but never overshoot it?

[LigH.de]

The exceeding is quite marginal. To be more certain, you may possibly need a working VBV regulation in conjunction with 2-pass, but this is not yet complete.

[Selur]

The exceeding is quite marginal.

if you are restricted to a medium size, 5kb/s overflow can still be a problem (5kbit/s underflow not so much).

you may possibly need a working VBV regulation in conjunction with 2-pass, but this is not yet complete.

since I normally use way higher VBV restrictions than the average bit rate I aim for, I'm not sure how VBV would help

---
Did a few encodes and for longer files the output seems to be always above the called datarate.

[x265]

The exceeding is quite marginal.

if you are restricted to a medium size, 5kb/s overflow can still be a problem (5kbit/s underflow not so much).

you may possibly need a working VBV regulation in conjunction with 2-pass, but this is not yet complete.

since I normally use way higher VBV restrictions than the average bit rate I aim for, I'm not sure how VBV would help

---
Did a few encodes and for longer files the output seems to be always above the called datarate.

I did a long 2 pass encode today, and the output was 98.6% of the target bit rate.

We're still working to improve all of our rate control algorithms. 2 pass is quite new, and you can expect ongoing improvements. I think that most customers/users would agree that a bit rate deviation of < 1.5% is not bad. If you need to stay under an absolute number, I suggest you adjust your target bit rate (reduce by 2%).

[Selur]

most customers/users would agree that a bit rate deviation of < 1.5% is not bad.

as long as it's an under- and not an overflow I would agree (besides the point that 1.5% can be quite a bit when keeping in mind that 4k+ video requires quite a bit of bandwidth)

2 pass is quite new, and you can expect ongoing improvements.

That's good since atm. it has problems with vbv restrictions and qp files.

[LigH.de]

@ x265:

I hope it will be possible to explain the difference between --psy-rd and --psy-rdoq briefly...

   --psy-rd <0..2.0>             Strength of psycho-visual rate distortion optimization, 0 to disable. Default 0.000000
   --psy-rdoq <0..2.0>           Strength of psycho-visual optimization in quantization, 0 to disable. Default 0.000000

It is not yet documented in reST, BTW.

[Selur]

@LigH: what version are you using?
1.2+305-66ed81577483 only shows:

Analysis:
--rd <0..6> Level of RD in mode decision 0:least....6:full RDO. Default 3
--psy-rd <0..2.0> Strength of psycho-visual rate distortion optimization, 0 to disable. Default 0.000000
--nr <integer> An integer value in range of 100 to 1000, which denotes strength of noise reduction. Default disabled
--[no-]tskip-fast Enable fast intra transform skipping. Default disabled
--[no-]early-skip Enable early SKIP detection. Default disabled
--[no-]fast-cbf Enable early outs based on whether residual is coded. Default disabled

no '--psy-rdoq',..

[LigH.de]

Built x265 1.2+349-8bab5275baed today.

[Selur]

I see, it came with: https://bitbucket.org/multicoreware/x265/commits/93a434014f5afa92aa62c70f1cc0fbe80b2b8404
"psy-rdoq is not yet functional - option is ignored"

+.. option:: --psy-rdoq <float>

+

+ Influence the rate distortion optimized quantization by favoring

+ higher energy in the reconstructed image. This generally improves

+ perceived visual quality at the cost of lower quality metric scores.

+ It only has effect on slower presets which use RDO Quantization

+ (rd-levels 4 and 5). Experimental

So basically, like all the rdo stuff: magic

[LigH.de]

Sounds like it should be used to alter quantization values already changed by --psy-rd even further. Well, I am curious for academic samples, "when it's done" ...

[Selur]

btw. did anyone ever try to detect differences between the different psy-rd settings in x265?

[LigH.de]

I once encoded the same videos with --psy-rd (0.0 .. 1.0) in steps of 0.2; if you are interested, look in my MediaFire directory (HEVC/samples).

[Selur]

did you come to any conclusions looking at the files? (assuming you did)

[x265]

We've checked in Psy-RDOQ. At this point, you should consider this feature highly experimental. We are performing tests to determine the scaling factor that is needed. Ongoing work includes Psy-RDOQ performance optimization.

As always, your test results (with full command line, --log 3 or 4 csv file with --SSIM on and ideally, a link to the publicly available source clip) are welcomed.

Tom