metadata not in resulting mp4 after ffmpeg NVenc transcoding

[hydra3333]

Hello.

If I do an ffprobe on an input file and then use the metadata (sometimes related to HDR) in the conversion, like this

"ffmpeg.exe" -hide_banner -v verbose -strict -1 -init_hw_device opencl=ocl:1.0 -filter_hw_device ocl -threads 0 -i "test01.h264.mp4" -threads 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_i np -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0 .5:cx=3:cy=3:ca=0.5,hwdownload" -c:v hevc_nvenc -pix_fmt yuv420p -profile:v main -level 5.1 -preset slow -rc vbr_hq -cq 26 -rc-lookahead 32 -spatial_aq 1 -x265-params "colormatrix=bt470bg" -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -threads 0 -movflags +faststart -y "test01.h265.mp4"

(eg the -x265-params "colormatrix=bt470bg") then when I ffprobe the output file I see that none of the metadata is there, neither copied from the input file nor from the commandline parameters which I assume are passed to libx265 and not the NV encoder.

I also tried to use the very interesting NVEencc, where one can set such parameters on the commandline, however I found that a number of encodes had a loss of audio sync whereas with ffmpeg the audio sync was maintained even if the metadata was missing. I'll find and post an example of the desync on the nvencc github to see if it can be fixed.

Anyway, any suggestions on how to get only a range of the colour-related metadata from the input file to the output file using ffmpeg per the above commandline ?

[poisondeathray]

yes -x265-params applies to libx265 , not nvenc

You can't currently with ffmpeg or nvencc ; it's not supported by nvidia sdk

But you can patch the metadata for nvenc hevc streams
https://github.com/SK-Hardwired/nv_hevc_hdr_patcher
https://www.videohelp.com/software/NVHSP


For the nvencc sync issue , was it using full hw decode ?

[hydra3333]

yes -x265-params applies to libx265 , not nvenc
You can't currently with ffmpeg or nvencc ; it's not supported by nvidia sdk

OK. Jolly, bother, and dash it all.

But you can patch the metadata for nvenc hevc streams
https://github.com/SK-Hardwired/nv_hevc_hdr_patcher
https://www.videohelp.com/software/NVHSP

OK, thank you.

For the nvencc sync issue , was it using full hw decode ?

Yes.

[Selur]

using ffmpeg directly using "color_primaries", "color_trc", "colorspace", should also work to set the signaling while not doing HDR.

color_primaries integer (decoding/encoding,video)

    Possible values:

    ‘bt709’

        BT.709 
    ‘bt470m’

        BT.470 M 
    ‘bt470bg’

        BT.470 BG 
    ‘smpte170m’

        SMPTE 170 M 
    ‘smpte240m’

        SMPTE 240 M 
    ‘film’

        Film 
    ‘bt2020’

        BT.2020 
    ‘smpte428’
    ‘smpte428_1’

        SMPTE ST 428-1 
    ‘smpte431’

        SMPTE 431-2 
    ‘smpte432’

        SMPTE 432-1 
    ‘jedec-p22’

        JEDEC P22 

color_trc integer (decoding/encoding,video)

    Possible values:

    ‘bt709’

        BT.709 
    ‘gamma22’

        BT.470 M 
    ‘gamma28’

        BT.470 BG 
    ‘smpte170m’

        SMPTE 170 M 
    ‘smpte240m’

        SMPTE 240 M 
    ‘linear’

        Linear 
    ‘log’
    ‘log100’

        Log 
    ‘log_sqrt’
    ‘log316’

        Log square root 
    ‘iec61966_2_4’
    ‘iec61966-2-4’

        IEC 61966-2-4 
    ‘bt1361’
    ‘bt1361e’

        BT.1361 
    ‘iec61966_2_1’
    ‘iec61966-2-1’

        IEC 61966-2-1 
    ‘bt2020_10’
    ‘bt2020_10bit’

        BT.2020 - 10 bit 
    ‘bt2020_12’
    ‘bt2020_12bit’

        BT.2020 - 12 bit 
    ‘smpte2084’

        SMPTE ST 2084 
    ‘smpte428’
    ‘smpte428_1’

        SMPTE ST 428-1 
    ‘arib-std-b67’

        ARIB STD-B67 

colorspace integer (decoding/encoding,video)

    Possible values:

    ‘rgb’

        RGB 
    ‘bt709’

        BT.709 
    ‘fcc’

        FCC 
    ‘bt470bg’

        BT.470 BG 
    ‘smpte170m’

        SMPTE 170 M 
    ‘smpte240m’

        SMPTE 240 M 
    ‘ycocg’

        YCOCG 
    ‘bt2020nc’
    ‘bt2020_ncl’

        BT.2020 NCL 
    ‘bt2020c’
    ‘bt2020_cl’

        BT.2020 CL 
    ‘smpte2085’

        SMPTE 2085 

color_range integer (decoding/encoding,video)

    If used as input parameter, it serves as a hint to the decoder, which color_range the input has. Possible values:

    ‘tv’
    ‘mpeg’

        MPEG (219*2^(n-8)) 
    ‘pc’
    ‘jpeg’

        JPEG (2^n-1) 

chroma_sample_location integer (decoding/encoding,video)

    Possible values:

    ‘left’
    ‘center’
    ‘topleft’
    ‘top’
    ‘bottomleft’
    ‘bottom’

see: https://www.ffmpeg.org/ffmpeg-all.html

[hydra3333]

Nice !
I didn't spot those, didn't look hard enough I guess.
Will give them a try.

edit 1: "while not doing HDR" Ah, I see, it doesn't have some HDR stuff like "master_display" "max_cll" and whatnot. Yet (is some fmpeg flag development for HDR in the works ?) ?

edit 2: yes and no. more no. https://trac.ffmpeg.org/ticket/7037 acknowledges an underlying issue (whilst not mentioning what appears to be something along the lines that the nvidia encoder doesn't "emit" the correct stuff in streams per above https://www.videohelp.com/software/NVHSP ... who knows, maybe HDR is an ongoing stuffup, unless nvidia fix their bit of it I guess and ffmpeg do theirs ?

[poisondeathray]

For the nvencc sync issue , was it using full hw decode ?

Yes.

I've noticed issues with full hw , both in ffmpeg and nvencc . Sometimes frames are dropped or duplicated . Current status is very unreliable in my experience. Yes, once in a while it might work on some streams without errors, but it's basically unusuable in my opinion. . This could be part of the explanation for the sync issues. Look at the frame count

Right now, software decoding is the only reliable method (or indexed with something like dgindexnv/dgdecodenv)




For the other ffmpeg switches -colorspace, etc... it's very dicey in my experience, and the behaviour has changed over time. I don't know what it is currently. But there are times where they actually change the data (ie not just flags) . Whereas something like -x264-params or -x265-params with colormatrix=blah are strictly metadata VUI options.

-pix_fmt yuv420p

When using ffmpeg nvenc (at least for AVC, not sure for HEVC, but I assume it applies) , nvenc expects NV12 (-pix_fmt nv12). You get chroma errors otherwise. Nvencc automatically converts to nv12

[hydra3333]

Ah. Well, now to stop/check/fix/restart my transcoding process ...

I wonder what's coming out of the yadif and unsharp bit and what the sequence is. I wonder if a high verbosity level will tell me.

[hydra3333]

Also wondering ...

Which pix_fmts to be specified for 10-bit HDR encoding, which apparently nvenc can do ?
eg p010le or p016le (I suppose, which is quicker in conversion ?)
edit: there's an interesting discussion here: https://ffmpeg.org/pipermail/ffmpeg-devel/2016-September/198666.html

And how does on tell nvenc to do 10-bit encoding ?

Encoder hevc_nvenc [NVIDIA NVENC hevc encoder]:
    General capabilities: delay 
    Threading capabilities: none
    Supported pixel formats: yuv420p nv12 p010le yuv444p p016le yuv444p16le bgr0 rgb0 cuda d3d11
hevc_nvenc AVOptions:
  -preset            <int>        E..V..... Set the encoding preset (from 0 to 11) (default medium)
     default                      E..V..... 
     slow                         E..V..... hq 2 passes
     medium                       E..V..... hq 1 pass
     fast                         E..V..... hp 1 pass
     hp                           E..V..... 
     hq                           E..V..... 
     bd                           E..V..... 
     ll                           E..V..... low latency
     llhq                         E..V..... low latency hq
     llhp                         E..V..... low latency hp
     lossless                     E..V..... lossless
     losslesshp                   E..V..... lossless hp
  -profile           <int>        E..V..... Set the encoding profile (from 0 to 4) (default main)
     main                         E..V..... 
     main10                       E..V..... 
     rext                         E..V..... 
  -level             <int>        E..V..... Set the encoding level restriction (from 0 to 186) (default auto)
     auto                         E..V..... 
     1                            E..V..... 
     1.0                          E..V..... 
     2                            E..V..... 
     2.0                          E..V..... 
     2.1                          E..V..... 
     3                            E..V..... 
     3.0                          E..V..... 
     3.1                          E..V..... 
     4                            E..V..... 
     4.0                          E..V..... 
     4.1                          E..V..... 
     5                            E..V..... 
     5.0                          E..V..... 
     5.1                          E..V..... 
     5.2                          E..V..... 
     6                            E..V..... 
     6.0                          E..V..... 
     6.1                          E..V..... 
     6.2                          E..V..... 
  -tier              <int>        E..V..... Set the encoding tier (from 0 to 1) (default main)
     main                         E..V..... 
     high                         E..V..... 
  -rc                <int>        E..V..... Override the preset rate-control (from -1 to INT_MAX) (default -1)
     constqp                      E..V..... Constant QP mode
     vbr                          E..V..... Variable bitrate mode
     cbr                          E..V..... Constant bitrate mode
     vbr_minqp                    E..V..... Variable bitrate mode with MinQP (deprecated)
     ll_2pass_quality              E..V..... Multi-pass optimized for image quality (deprecated)
     ll_2pass_size                E..V..... Multi-pass optimized for constant frame size (deprecated)
     vbr_2pass                    E..V..... Multi-pass variable bitrate mode (deprecated)
     cbr_ld_hq                    E..V..... Constant bitrate low delay high quality mode
     cbr_hq                       E..V..... Constant bitrate high quality mode
     vbr_hq                       E..V..... Variable bitrate high quality mode
  -rc-lookahead      <int>        E..V..... Number of frames to look ahead for rate-control (from 0 to INT_MAX) (default 0)
  -surfaces          <int>        E..V..... Number of concurrent surfaces (from 0 to 64) (default 0)
  -cbr               <boolean>    E..V..... Use cbr encoding mode (default false)
  -2pass             <boolean>    E..V..... Use 2pass encoding mode (default auto)
  -gpu               <int>        E..V..... Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on. (from -2 to INT_MAX) (default any)
     any                          E..V..... Pick the first device available
     list                         E..V..... List the available devices
  -delay             <int>        E..V..... Delay frame output by the given amount of frames (from 0 to INT_MAX) (default INT_MAX)
  -no-scenecut       <boolean>    E..V..... When lookahead is enabled, set this to 1 to disable adaptive I-frame insertion at scene cuts (default false)
  -forced-idr        <boolean>    E..V..... If forcing keyframes, force them as IDR frames. (default false)
  -spatial_aq        <boolean>    E..V..... set to 1 to enable Spatial AQ (default false)
  -temporal_aq       <boolean>    E..V..... set to 1 to enable Temporal AQ (default false)
  -zerolatency       <boolean>    E..V..... Set 1 to indicate zero latency operation (no reordering delay) (default false)
  -nonref_p          <boolean>    E..V..... Set this to 1 to enable automatic insertion of non-reference P-frames (default false)
  -strict_gop        <boolean>    E..V..... Set 1 to minimize GOP-to-GOP rate fluctuations (default false)
  -aq-strength       <int>        E..V..... When Spatial AQ is enabled, this field is used to specify AQ strength. AQ strength scale is from 1 (low) - 15 (aggressive) (from 1 to 15) (default 8)
  -cq                <float>      E..V..... Set target quality level (0 to 51, 0 means automatic) for constant quality mode in VBR rate control (from 0 to 51) (default 0)
  -aud               <boolean>    E..V..... Use access unit delimiters (default false)
  -bluray-compat     <boolean>    E..V..... Bluray compatibility workarounds (default false)
  -init_qpP          <int>        E..V..... Initial QP value for P frame (from -1 to 51) (default -1)
  -init_qpB          <int>        E..V..... Initial QP value for B frame (from -1 to 51) (default -1)
  -init_qpI          <int>        E..V..... Initial QP value for I frame (from -1 to 51) (default -1)
  -qp                <int>        E..V..... Constant quantization parameter rate control method (from -1 to 51) (default -1)
  -weighted_pred     <int>        E..V..... Set 1 to enable weighted prediction (from 0 to 1) (default 0)

[hydra3333]

Ah, I suppose to use hevc_nvenc that'd be

• for 8bit, pix_fmts=nv12 with profile main
• for 10bit HDR, pix_fmts=yuv444p16le and just use profile main10

[hydra3333]

-pix_fmt yuv420p

When using ffmpeg nvenc (at least for AVC, not sure for HEVC, but I assume it applies) , nvenc expects NV12 (-pix_fmt nv12). You get chroma errors otherwise. Nvencc automatically converts to nv12

Using nv12 this

"ffmpeg.exe" -hide_banner -v verbose -strict -1 -init_hw_device opencl=ocl:1.0 -filter_hw_device ocl -threads 0 -i "test01.mp4.mp4" -threads 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_i np -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0 .5:cx=3:cy=3:ca=0.5,hwdownload" -c:v hevc_nvenc -pix_fmt nv12 -profile:v main -level 5.1 -preset slow -rc vbr_hq -cq 26 -rc-lookahead 32 -spatial_aq 1 -color_range tv -colorspace bt470bg -color_primaries bt470bg -color_trc gamma28 -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -threads 0 -movflags +faststart -y "G:\HDTV\000-Pi-make-hevc\converted_hevc\mp4library\to_convert1\Classic Documentaries\test01.mp4.h265.mp4"

yields error

[h264 @ 0000015b12e0e880] Reinit context to 720x576, pix_fmt: yuv420p
[graph_1_in_0_1 @ 0000015b1408abc0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
[format_out_0_1 @ 0000015b1408af80] auto-inserting filter 'auto_resampler_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_1'
[auto_resampler_0 @ 0000015b1408b3c0] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz
[graph 0 input from stream 0:0 @ 0000015b14348f00] w:720 h:576 pixfmt:yuv420p tb:1/90000 fr:25/1 sar:71/50 sws_param:flags=2
[hwdownload @ 0000015b1426e200] Invalid output format nv12 for hwframe download.
[Parsed_hwdownload_3 @ 0000015b14348e00] Failed to configure output pad on Parsed_hwdownload_3
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
[libfdk_aac @ 0000015b12e0dac0] 2 frames left in the queue on closing
Conversion failed!

It also fails if I instead use this at the end of the filter_complex

,format=pix_fmts=nv12

whereas this "works" with yuv420p

"ffmpeg.exe" -hide_banner -v verbose -strict -1 -init_hw_device opencl=ocl:1.0 -filter_hw_device ocl -threads 0 -i "G:\HDTV\000-Pi-make-hevc\test01.mp4.mp4" -threads 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_i np -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0 .5:cx=3:cy=3:ca=0.5,hwdownload" -c:v hevc_nvenc -pix_fmt yuv420p -profile:v main -level 5.1 -preset slow -rc vbr_hq -cq 26 -rc-lookahead 32 -spatial_aq 1 -color_range tv -colorspace bt470bg -color_primaries bt470bg -color_trc gamma28 -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -threads 0 -movflags +faststart -y "G:\HDTV\000-Pi-make-hevc\converted_hevc\mp4library\to_convert1\Classic Documentaries\test01.mp4.h265.mp4"

Can someone please point out where I'm going wrong ?


edit: Well there you go. this worked

"ffmpeg.exe" -hide_banner -v verbose -strict -1 -init_hw_device opencl=ocl:1.0 -filter_hw_device ocl -threads 0 -i "G:\HDTV\000-Pi-make-hevc\test01.mp4.mp4" -threads 0 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_i np -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0 .5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv 420p" -c:v hevc_nvenc -pix_fmt nv12 -profile:v main -level 5.1 -preset slow -rc vbr_hq -cq 26 -rc-lookahead 32 -spatial_aq 1 -color_range tv -colorspace bt470bg -color_primaries bt470bg -color_trc gamma28 -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000 -threads 0 -movflags +faststart -y "G:\HDTV\000-Pi-make-hevc\converted_hevc\mp4library\to_convert1\Classic Documentaries\test01.mp4.h265.mp4"

[poisondeathray]

And how does on tell nvenc to do 10-bit encoding ?

probably -pix_fmt p010le . I don't have a pascal card to test . I don' t know if you get problems or artifact with one or the other, or if one conversion is slower because of a certain pixel format. After all, yuv420p was claimed as "supported" by h264_nvenc, and you get artifacts . Feel free to do some qualitative and speed tests if you have time

For the 8bit AVC case, the yuv420p vs. nv12 artifact issue looks like the screenshots in this posts
https://forum.videohelp.com/threads/383320-So-what-s-the-status-on-hardware-accelerate...ng#post2484298
https://forum.videohelp.com/threads/383320-So-what-s-the-status-on-hardware-accelerate...ng#post2484301

[hydra3333]

OK thanks,what a great thread you linked to. I did spot the examples although I can't see to see them now. The ones below roof tiles. Anyway, thank you for that.

At face value it seems to be a combination of profile main10 and the input pix_fmt to bump 10bit hevc encoding into action. Although there seems to be a fair bit of older and therefore currently-mis-information floating around, I did also read somewhere that 420 somehow causes it not to kick in whereas 444 does. Who knows, worth a few tests.

I also keep reading that nvenc doesn't encode interlaced whereas this nvidia's response on the nvidia developer forum to this user query

I tested Cuda 8.0, version nvidia from 367.42 to 375.20, ffmpeg 3.2 and ffmpeg git version in today. (--enable-cuda --enable-cuvid --enable-rpath --enable-nvenc)
run ffmpeg with options "-flags:v +ildct -top 1"
My picture is bad, showing this same what in post #1.

says Posted 01/10/2017 04:46 AM

This issue is root-caused and fix will be available in our next driver branch (due out in February 2017). Thank you for reporting the issue.

with the user response

This issue has been fixed in the new 375.39 driver. Thanks!

which is a few versions prior to the current driver ... so interlaced encoding also appears to be worth a few tests with the latest nvidia driver.

[poisondeathray]

I also keep reading that nvenc doesn't encode interlaced whereas this nvidia's response on the nvidia developer forum to this user query

You referring to interlaced HEVC ? Officially it's not even fully supported by HEVC (there is partial support in metadata). The early drafts were "progressive only, interlace is dead." Not all decoders are guaranteed to support interlace HEVC, so playback might be problematic. Not sure about nvenc hevc, because I don't have a pascal card

NVEnc does support AVC, PAFF . (it's one of the "free" ways to get AVC PAFF encoding , if you have an nvidia card. x264 only does MBAFF, which is actually better, but there are some rare cases that you might need PAFF) . In contrast, interlace as PAFF and MBAFF are officially supported in AVC

[hydra3333]

My bad
https://devtalk.nvidia.com/default/topic/976088/video-codec-sdk/ffmpeg-h264_nvenc-interlaced-encode/
h.264 interlaced which I misread as h.265.

They don't clarify, so it must be PAFF I guess. Worth a try to h.264 to see with mediainfo and ffprobe, but since i'm looking to move to h.265 it's a tad moot for me.

As ffmpeg currently doesn't appear to support the full set of HDR flags (as you mentioned, extra tools are required which I'm not altogether keen on due to all the issues I've had over time with separated video/audio streams and lost audio sync) HDR is also a bit moot. Yes there's NVEncc which is lovely but I've had some issues with it including older clips transcoded into something 5mm wide (yes hw decoding). I hope the ffmpeg developers recognise HDR is almost here in consumer grade cameras etc and are willing to add full support for it.

[poisondeathray]

Yes h.264 nvenc interlace works - it's definitely PAFF. As I mentioned above, it's one of the free (if you have a nvidia card) ways to get PAFF (Interlace) encoding. x264 can't do it. (MBAFF is much better type of "interlace", more efficient)

Use software x265 if you need HDR .

NVEnc HEVC doesn't have b-frames either . From what I've seen, the compression quality isn't very good. It's probably the worst HEVC encoder. But probably the fastest too

[hydra3333]

OK. Thank you. Yes, worst is relative to need, for a 4Tb set of mediocre home clips maybe 300fps transcoding is "ok enough" for those rather than wait forever for 5-10fps with x265.

I was changing my workflow to be "more compatible" with a Chromecast Ultra (I had a few audio and video problems) and "getting ready" for HDR in that, so I suppose I could just leave them mainly as h.264 larger files without the extra hevc compression benefit and buy more disks.

[poisondeathray]

so I suppose I could just leave them mainly as h.264 larger files without the extra hevc compression benefit and buy more disks.

What extra hevc compression benefit ? It's NVenc HEVC

[hydra3333]

Ah. Yes. Now that you point it out, I may as well do h.264.

[poisondeathray]

Ah. Yes. Now that you point it out, I may as well do h.264.

Not just any h.264. x264

There is a definite compression advantage over nvenc h264 .

But nvenc hevc actually scores worse than x264 in objective and subjective tests. Even at UHD resolutions, one scenario where hevc should hold a massive advantage (largely due to larger "block" CTU sizes), nvenc hevc somehow is worse. It's only "hevc" in name, but underperforms in terms of compression ratio / quality.

If compatibility wasn't an issue (not looking to play on say, some older devices), and you were looking mostly at speed it would be a good alternative.