AMD GPU Transcoding MJPEG to HEVC provides a skewed split screen video.

[rajhlinux]

Hello,

I am trying to use AMD GPU for transcoding MJPEG to HEVC using VA-API.

Using this command:

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -c:v mjpeg -f v4l2 -input_format mjpeg -framerate 30 -video_size 1920x1080 -i /dev/video0 -vf 'scale_vaapi=format=nv12' -c:v hevc_vaapi -map 0:a? -map 0:s? -c:s copy -rc_mode 1 -qp 25 -map_chapters 0 output.mp4 -y

And it seems to work fine, no skipped frames, smooth transcoding, only 2-3% CPU usage. However the video output file is completely skewed.

For some reason FFMPEG splits the video into two pieces where one half of the video is the actually the proper video and the other half is all skewed up.

Note: The camera is a CCTV camera, which is using it's infrared LEDs to view in the dark room. So it is in greyscale color due to the camera and not any issues based on transcoding.

Here is a picture of the video output:
https://imgur.com/OuMlidj

Here is the output video file:
https://www.youtube.com/watch?v=XCC4JDffTxM

Any clue why this is happening?

I have AMD AMF and VAAPI encoders working. I like VAAPI since it is able to do HEVC encoding.

OS: Arch Linux

GPU: AMD Radeon Pro WX 2100.

$ lspci -k | grep -A 2 -E "(VGA|3D)"

bash: If: command not found...
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 2100]
        Subsystem: Dell Device 0b0c
        Kernel driver in use: amdgpu

Thanks for any advice.

[pandy]

It looks like incompatible color data format between MJPEG and HEVC - perhaps you should use something else than NV12 or convert MJPEG explicitly to NV12 and later to HEVC on AMD AMF framework (at beginning perhaps using two ffmpeg instance in pipe may help to isolate root cause for issue source)

Perhaps try to querry 'hevc_vaapi' for supported color data formats (not using linux so i hope windows command 'ffmpeg -h encoder=hevc_vaapi' will work similar in linux).

Using AMD GPU is not the quality best but if your goal is to reduce CPU utilization then acceptable.

[rajhlinux]

Hello, Thank You for replying.

I will try that command out.

EDIT:

This is the command I used:

$ ffmpeg -h encoder=hevc_vaapi

ffmpeg version n5.0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.1.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-shared --enable-version3
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
Encoder hevc_vaapi [H.265/HEVC (VAAPI)]:
    General capabilities: dr1 delay hardware
    Threading capabilities: none
    Supported hardware devices: vaapi
    Supported pixel formats: vaapi
h265_vaapi AVOptions:
  -low_power         <boolean>    E..V....... Use low-power encoding mode (only available on some platforms; may not support all encoding features) (default false)
  -idr_interval      <int>        E..V....... Distance (in I-frames) between IDR frames (from 0 to INT_MAX) (default 0)
  -b_depth           <int>        E..V....... Maximum B-frame reference depth (from 1 to INT_MAX) (default 1)
  -rc_mode           <int>        E..V....... Set rate control mode (from 0 to 6) (default auto)
     auto            0            E..V....... Choose mode automatically based on other parameters
     CQP             1            E..V....... Constant-quality
     CBR             2            E..V....... Constant-bitrate
     VBR             3            E..V....... Variable-bitrate
     ICQ             4            E..V....... Intelligent constant-quality
     QVBR            5            E..V....... Quality-defined variable-bitrate
     AVBR            6            E..V....... Average variable-bitrate
  -qp                <int>        E..V....... Constant QP (for P-frames; scaled by qfactor/qoffset for I/B) (from 0 to 52) (default 0)
  -aud               <boolean>    E..V....... Include AUD (default false)
  -profile           <int>        E..V....... Set profile (general_profile_idc) (from -99 to 255) (default -99)
     main            1            E..V.......
     main10          2            E..V.......
     rext            4            E..V.......
  -tier              <int>        E..V....... Set tier (general_tier_flag) (from 0 to 1) (default main)
     main            0            E..V.......
     high            1            E..V.......
  -level             <int>        E..V....... Set level (general_level_idc) (from -99 to 255) (default -99)
     1               30           E..V.......
     2               60           E..V.......
     2.1             63           E..V.......
     3               90           E..V.......
     3.1             93           E..V.......
     4               120          E..V.......
     4.1             123          E..V.......
     5               150          E..V.......
     5.1             153          E..V.......
     5.2             156          E..V.......
     6               180          E..V.......
     6.1             183          E..V.......
     6.2             186          E..V.......
  -sei               <flags>      E..V....... Set SEI to include (default hdr)
     hdr                          E..V....... Include HDR metadata for mastering display colour volume and content light level information
  -tiles             <image_size> E..V....... Tile columns x rows

Thanks.

[pandy]

Please read this: https://trac.ffmpeg.org/wiki/Hardware/VAAPI

[rajhlinux]

I have checked that link and none of the code works for transcoding for webcam.

The only code that works from that link are the following code:

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -c:v mjpeg -f v4l2 -input_format mjpeg -framerate 30 -video_size 1920x1080 -i /dev/video0 -vf 'scale_vaapi=format=nv12' -c:v hevc_vaapi -map 0:a? -map 0:s? -c:s copy -rc_mode 1 -qp 25 -map_chapters 0 output.mp4 -y

I have literally tried hundreds of different combinations of implementing transcoding command combinations from that link and I found that I must add "-vf scale_vaapi=format=nv12" and also "-c:v mjpeg" or "-f v4l2 -input_format mjpeg" to allow the FFMPEG to actually do something in transcoding, in which it provides a skewed video file.

It seems 'scale_vaapi=" is the main reason why FFMPEG actually allows the transcoding to take place, it is also the reason why the video is skewed.

I can also run this command, however this command skips the decoding and it only encodes, it also takes CPU resource up to 20%, the command below is also taken from the link you have provided and it simply captures RAW stream and just encodes it:

ffmpeg -vaapi_device /dev/dri/renderD128 -f v4l2 -video_size 1920x1080 -i /dev/video0 -vf 'format=nv12,hwupload' -c:v h264_vaapi output.mp4

Thanks

[pandy]

As i have no machine with Linux and AMD GPU so i can share only some thoughts... From what i can observe seems there is some issue with pixel format - Luminescence seem to occupy half of screen, rest is garbage data.

'hwupload' seem to be recommended way to promote video data inside GPU buffer - not sure about 'scale_vaapi' command...

Perhaps your camera provide only luminance plane (Y) and it is up-converted in a dumb way to NV12 where Cb and Cr samples are substituted by some garbage data...

Try to isolate issue - split your command line on two, also adding '-report' command to ffmpeg may be useful.

[rajhlinux]

Thank You for your insigts.

Would like to mention that my camera is a CCTV camera which outputs CVI analog video, this video is then converted to HDMI using a special converter box, then the HDMI output is attached to a USB 3.0 capture card so that the CCTV video feed can be viewed on the PC.

I am not sure how to split the command line into two.

I have asked the question to ffmpeg mailing list.

[pandy]

By splitting your command line on two separate actions i mean separate encoding from capture process - for this moment you need to debug/investigate where is the issue as documentation for ffmpeg on AMD GPU usage with vaapi is quite vague...

Perhaps you can capture raw data from your camera to file and in second step use those data to find working GPU encode settings.

Dig a bit on vaapi and it seem there is many different pixel formats supported there but all of them are hidden/unknown so it may be that you push NV12 and it works from GPU perspective but fail on capture part - so you end with miss aligned data as such this may explain provided snapshot.

Btw - your setup is to be honest quite odd - not sure if there is any sense to use SD camera with HD capture/upconverter (at least this how i understand description of your setup configuration).

Btw 2 - 'scale_vaapi' command may work as magic cookie where wrong pixel format is marked as good but this may be the main issue...

[rajhlinux]

Hello pandy,

So my setup is a bit different. I have attached a picture exactly how everything works:


[Attachment 66149 - Click to enlarge]
https://forum.videohelp.com/images/imgfiles/Vmy9Wi4.jpg

The capture card takes in the HDMI signal and outputs it to USB 3.0 so I can view the camera stream using V4L2.
It outputs the stream to MJPG.

Here are some details about the USB 3.0 capture card:

$ v4l2-ctl --all --device /dev/video0:

Driver Info:
        Driver name      : uvcvideo
        Card type        : USB3.0 HD Video Capture: USB3.0
        Bus info         : usb-0000:04:00.0-2
        Driver version   : 5.18.12
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : uvcvideo
        Model            : USB3.0 HD Video Capture: USB3.0
        Serial           : HU123450
        Bus info         : usb-0000:04:00.0-2
        Media version    : 5.18.12
        Hardware revision: 0x00000328 (808)
        Driver version   : 5.18.12
Interface Info:
        ID               : 0x03000002
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : USB3.0 HD Video Capture: USB3.0
        Function         : V4L2 I/O
        Flags            : default
        Pad 0x01000007   : 0: Sink
          Link 0x02000010: from remote pad 0x100000a of entity 'Processing 3' (Video Pixel Formatter): Data, Enabled, Immutable
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height      : 1920/1080
        Pixel Format      : 'MJPG' (Motion-JPEG)
        Field             : None
        Bytes per Line    : 0
        Size Image        : 2073600  [Megapixels: 2.1 MP (2,073,600 pixels)]
        Colorspace        : sRGB
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Default (maps to Full Range)
        Flags             :
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 1920, Height 1080
        Default     : Left 0, Top 0, Width 1920, Height 1080
        Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 1920, Height 1080, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags:
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 60.000 (60/1)
        Read buffers     : 0

$ v4l2-ctl --list-formats-ext --device /dev/video0

ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'MJPG' (Motion-JPEG, compressed)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 960x540
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 800x450
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 640x360
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)

So with this in mind, how can I output to RAW video stream data?

Thanks.