Confused by the output Handbrake is giving me for a tricky excessive encode

[Cyber Akuma]

I have a videogame demo I wanted to record footage of. A big part of the reason was that on top of just simply liking the demo and it’s visuals, the special effects and visuals of this game particularly looked difficult to encode, almost as if it was made to be hard to capture, so I wanted to try, and then encode a version just to keep locally as well as an excessively exaggerated encode for the purposes of uploading to YouTube mainly to see if it would look good there since it would be much easier to share it that way than as a file.

To try to avoid as much quality and frame loss as possible during my initial capture, I set OBS to it’s lossless profile with color format set to RGB. By my calculations a standard playthrough of the demo for 50-60 minutes at 1080p 60FPS would have been about a 380-420GB capture, and I was using a 500GB SSD for it.

Seems my calculations were slightly off and the file ended up being smaller at 286GB at 41 minutes, quality was pretty good and only 5 frames were dropped according to the log. Just a shame that I did poorly on the last level, if I had beaten the demo faster I might have gotten the filesize down to below 256GB and just uploaded that straight to YouTube while trying to figure out how to best encode a local version.

Anyway, so I fired up Handbrake, and hearing that YouTube tends to prefer x264, while using other codecs tend to result in being re-encoded worse, I made a profile specifically designed to try to encode it as excessively in x264 as possible:

https://forum.videohelp.com/images/imgfiles/FxpSolR.png

(Yes I know that’s just a few seconds, it was a test video in that screenshot)

After about 4:30 hours I ended up with….. a surprisingly small 5.76GB file. Quality was not really too great, it was passable, but scenes with fast movement and many bright particles and effects took a huge hit on the quality.

Curious as to why the file ended up being so small on what is essentially “lossless” settings for h264, I tried looking into what else I could try. NVENC 264 was an option, but I assumed that NVENC was lower quality intended for livestreaming more than encoding pre-recorded captures. Attempting to Google this was unhelpful as every result I could find was about CPU vs NVENC for livestreaming, not for encoding pre-recorded video. I also read that apparently RTX series cards have a better version called HENC (Though it’s not clear if this just simply supports HDR/10bit, or is actually better quality) but I only have a GTX card. I tried NVENC anyway with the same settings, and about 30-40 min later I was surprised to see it spit out a file that was almost twice as large at 11.3GB.

This file was certainly better quality, but still had flaws during such scenes I mentioned.

Not really sure what to try and since the game did have a lot of color gradients, I figured attempting another encode at h264 10bit would not hurt. From what I Googled, this would normally result in a file about 1.25 times bigger than 8 bit, which made perfect sense. However, when I came back a few hours after having started the encode I noticed that the encoder had paused at about the 40% mark because it had run out of space, even though there was more than enough space for even 4x the size of the NVENC encode.

I restarted it setting it to the only drive I had with plenty of space to spare, the same SSD the lossless file was on. This seemed to slow it down dramatically, estimating a 22-24 hour encode. While I know I was reading and writing to the same drive now, it was a separate read and write operation to a SSD, and I figured there was no way a single consumer grade SSD was saturating a SATA III port and it was likely very off in it’s estimate so I left it to complete.

….. 23 hours later it spits out a massive 72.5GB file. While yes, this file was of good quality finally, I am utterly confused why it’s nearly seven times bigger than the NVENC 264 and nearly 14 times bigger than the h.x264 encode.

I wondered if maybe there was some kind of bug or something in this version of either Handbrake or the encoder with the settings being so high that it disregards them for some reason. So I tried setting the encoder preset from Placebo to Very Slow…. same filesize.

I tried setting the CRF from 0 to 0.5…. same filesize.

I tried setting both the preset to Very Slow and CRF to 2…. same filesize.

I am just confused at this point. Why is a supposedly “lossless” version of an encode clearly encoding at not just lossy but apparently low quality settings? Why did NVENC encode the same video with the same codec and same settings at twice the filesize? Why did a 10 bit version end up being 14 times larger than the 8 bit version of the same codec instead of closer to 1.25 times? And I am confused just how to best create both a decently encoded version to store locally as well as a version that’s as excessive as reasonably possible but still under YouTube’s 256GB limit (I am not expecting said file to be anywhere near 256GB mind you) just as a temp file to upload to YouTube.

[davexnet]

Provide the mediainfo report (text view) of the both the source and your output file

[Cyber Akuma]

Source:

Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 286 GiB
Duration : 36 min 29 s
Overall bit rate : 1 124 Mb/s
Writing application : Lavf58.29.100

Video
ID : 0
Format : RGB
Codec ID : ULRG
Codec ID/Info : Ut Video Lossless Codec
Codec ID/Hint : Ut Video
Duration : 36 min 20 s
Source duration : 36 min 29 s
Bit rate : 1 012 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : RGB
Bit depth : 8 bits
Bits/(Pixel*Frame) : 8.135
Stream size : 257 GiB (90%)

Audio
ID : 1
Format : PCM
Format settings : Little / Signed
Codec ID : 1
Duration : 36 min 29 s
Bit rate mode : Constant
Bit rate : 1 536 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Bit depth : 16 bits
Stream size : 401 MiB (0%)
Alignment : Aligned on interleaves
Interleave, duration : 21 ms (1.28 video frame)

8 bit CPU Encode:

Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42 (isom/iso2/avc1/mp41)
File size : 5.77 GiB
Duration : 41 min 8 s
Overall bit rate : 20.1 Mb/s
Encoded date : UTC 2021-06-28 11:59:31
Tagged date : UTC 2021-06-28 11:59:31
Writing application : HandBrake 1.3.3 2020061300

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L4
Format settings : CABAC / 4 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 4 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 41 min 8 s
Bit rate : 19.7 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 60.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.159
Stream size : 5.67 GiB (98%)
Writing library : x264 core 157 r2935 545de2f
Encoding settings : cabac=1 / ref=4 / deblock=1:0:0 / analyse=0x1:0x131 / me=tesa / subme=11 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=0 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=24 / lookahead_threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=600 / keyint_min=60 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=20000 / vbv_bufsize=25000 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00
Encoded date : UTC 2021-06-28 11:59:31
Tagged date : UTC 2021-06-28 11:59:31
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Codec configuration box : avcC

Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : mp4a-40-2
Duration : 41 min 8 s
Bit rate mode : Constant
Bit rate : 319 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 93.8 MiB (2%)
Title : Stereo
Default : Yes
Alternate group : 1
Encoded date : UTC 2021-06-28 11:59:31
Tagged date : UTC 2021-06-28 11:59:31

10 bit CPU Encode:

ormat : Matroska
Format version : Version 2
File size : 72.9 GiB
Duration : 41 min 8 s
Overall bit rate : 254 Mb/s
Encoded date : UTC 2021-06-30 16:36:22
Writing application : mkvmerge v42.0.0 ('Overtime') 64-bit
Writing library : libebml v1.3.10 + libmatroska v1.5.2

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High 10@L5.1
Format settings : CABAC / 16 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 16 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 41 min 8 s
Bit rate : 252 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 60.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 2.027
Stream size : 72.5 GiB (99%)
Writing library : x264 core 157 r2935 545de2f
Encoding settings : cabac=1 / ref=16 / deblock=1:0:0 / analyse=0x3:0x133 / me=tesa / subme=11 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=24 / lookahead_threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=600 / keyint_min=60 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=0.0 / qcomp=0.60 / qpmin=0 / qpmax=81 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

Audio
ID : 2
Format : PCM
Format settings : Little / Signed
Codec ID : A_PCM/INT/LIT
Duration : 41 min 8 s
Bit rate mode : Constant
Bit rate : 1 536 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 32.000 FPS (1500 SPF)
Bit depth : 16 bits
Stream size : 452 MiB (1%)
Default : Yes
Forced : No

I no longer have the NVENC encode, I also put the original PCM audio back in the 10bit encode.

[davexnet]

I wonder if the "main" profile (8 bit) is constraining the encode in some way. Did you try "high" ?

Or use vdub2 and set compression to FFv1 chroma 422 or 420

[Cyber Akuma]

I am about to try a high encode. I got suggestions to try setting the encoder tuning to film, the profile to high, and that the encoder level of 4.0 was also likely limiting me.... not sure if I should set that to the highest of 5.2 or to auto.

(Also, I guess for the current temp big encode intended to be uploaded to YouTube, I should use FLAC over AAC, still need to decide how to do the smaller encode I plan to keep locally)

How big would a FFv1 file be? Isn't that similar to Huffyuv or so?

[davexnet]

I am about to try a high encode. I got suggestions to try setting the encoder tuning to film, the profile to high, and that the encoder level of 4.0 was also likely limiting me.... not sure if I should set that to the highest of 5.2 or to auto.

(Also, I guess for the current temp big encode intended to be uploaded to YouTube, I should use FLAC over AAC, still need to decide how to do the smaller encode I plan to keep locally)

How big would a FFv1 file be? Isn't that similar to Huffyuv or so?

"auto" on the level would be best. FFv1 is quite a bit smaller that Huffy, perhaps use a small section and do some test encodes

[Cyber Akuma]

Hmm, if I can bring it down to below 256GB I guess I could at least upload that to YouTube while I figure out what to do with the local encode. Never used Vdub2 (or even knew it existed, I gave up on anything vdub as outdated and dead long ago)

Should I be using an YUV color format though if it was recorded in RGB? Not sure what colorspace to set FFv1 to if the source is RGB.

[davexnet]

Vdub2 is recent, well worth a look

[Cyber Akuma]

Can you give any suggestions for what I should do about color space I mentioned above?

[jagabo]

None of that will effect quality at CRF=0 (as in your screenshot). But your mediainfo reports for one of them shows CRF=1. Still, at 1 there shouldn't be much difference between the source and the x264 encoded video. With these exceptions:

Even if the x264 compression is lossless, the conversion from RGB to YUV 4:2:0 (before the compression) will lose some precision and cause some posterization.

The conversion from RGB to YUV 4:2:0 will blur colors at edges and very small colored object will lose saturation. If you have small colored text it will become blurry. See: https://forum.videohelp.com/threads/319360-DVD-LAB-PRO-color-map#post1977264

I recommend you upload a smallish (<500 MB) sample of your lossless source and that same clip encoded with your settings. Then we can analyze what's going on.

<edit>

I just checked with handbrake: set Encoder Profile and Encoder Level to Auto. You'll get true lossless encoding. Otherwise those settings will change the CRF from 0 to 1 and limit bitrate.

</edit>

[Cyber Akuma]

It was tricky to get a 500-ish MB piece of the capture, I tried to find a scene that has a lot going on within that, though the game had many scenes with different effects in different levels:

https://mega.nz/file/tc4QSQAJ#eEQHiQDgIPGKe52wNBJFzKYlLtsES4G4f0i5DOUW4zk

Also I tried encoding it with FFv1, but it wound up just barely being larger than 256GB.... though, does YouTube even accept videos in FFv1 codec? I was just told that it does not, but I could not verify this.

[davexnet]

It was tricky to get a 500-ish MB piece of the capture, I tried to find a scene that has a lot going on within that, though the game had many scenes with different effects in different levels:

https://mega.nz/file/tc4QSQAJ#eEQHiQDgIPGKe52wNBJFzKYlLtsES4G4f0i5DOUW4zk

Also I tried encoding it with FFv1, but it wound up just barely being larger than 256GB.... though, does YouTube even accept videos in FFv1 codec? I was just told that it does not, but I could not verify this.

It depends on what options you set in the codec. For 8-bit @ 422 chroma I get approx 10:1 size reduction RGB > FFV1

[jagabo]

Handbrake at crf=0, Profile=Auto, Level=Auto, x264 preset=slow gave a 233 MB file. About the only difference you'll see is in the red text at the bottom left of the frame. The burring of the chroma is caused by the conversion to YUV 4:2:0 before the lossless compression. For uploading to youtube that doesn't matter -- even if you upload RGB or YUV 4:4:4 youtube will convert to YUV 4:2:0 when it recompresses your video.

At the placebo setting the file was a little smaller, 224 MB. At the slow preset but with crf=12 (visually near lossless, even looking at enlarged still frames) the video was 92 MB. I doubt you'd see any difference between crf12 and crf0 after youtube has mangled it.

[Cyber Akuma]

By the way, about uploading to YouTube. I had someone tell me that they tend to upscale their 1080p videos to 4K before uploading them to YouTube because apparently that makes YouTube use some sort of better quality compression algorithm on them even for the 1080p version? Any idea if that is true? Though Handbrake seems to have no upscaling features and the ones by default on vdub2 are pretty lackluster, especially since I only really care about the native 1080p version.

[jagabo]

Handbrake hasn't let you upscale video for quite a while. But I happened to look at the latest nightly build about a week ago. I noticed an upscale option. I didn't try it so I don't know anything else about it.

[dietboby]

I did give it a try, I am using CMD / CLI and Rigaya's Nvenc with an RTX2060 and a GTX 1070.

Your file - MediInfo readings

General
Complete name                            : I:\Lossless Capture Sample.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 551 MiB
Duration                                 : 5 s 184 ms
Overall bit rate                         : 892 Mb/s

Video
ID                                       : 0
Format                                   : RGB
Codec ID                                 : ULRG
Codec ID/Info                            : Ut Video Lossless Codec
Codec ID/Hint                            : Ut Video
Duration                                 : 5 s 183 ms
Bit rate                                 : 890 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 60.000 FPS
Color space                              : RGB
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 7.157
Stream size                              : 550 MiB (100%)

Audio
ID                                       : 1
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 5 s 184 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 972 KiB (0%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 21  ms (1.28 video frame)

After using NVENC

General
Unique ID                                : 11135879146727727871481244858456056223 (0x860B13B4412BAA5A8DCACA82ADDA19F)
Complete name                            : I:\Lossless Capture Sample.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 68.7 MiB
Duration                                 : 5 s 205 ms
Overall bit rate                         : 111 Mb/s
Writing application                      : NVEncC (x64) 5.34
Writing library                          : Lavf58.49.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L5
Format settings                          : CABAC / 8 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 8 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 5 s 205 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : Yes
Forced                                   : No

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 5 s 205 ms
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Default                                  : Yes
Forced                                   : No

Ive attached the files encoded via RTX 2060 here.

Commands used for NVENC.

--codec avc -u quality --cqp 20:23:25 --lookahead 32 --bframes 4 --ref 8 --bref-mode each --aq --aq-temporal --aq-strength 0 --vpp-smooth --mv-precision Q-pel --audio-codec aac --audio-bitrate 128 --audio-stream :stereo

Full command line - calling NVEncC and providing input file name, commands used and output file name.

"X:\NVencC\NVEncC64.exe" --avsw -i "Lossless Capture Sample.avi" --codec avc -u quality  --cqp 24:29:30 --lookahead 32 --bframes 4 --ref 8 --bref-mode each --aq --aq-temporal --aq-strength 0 --vpp-smooth --mv-precision Q-pel --audio-codec aac --audio-bitrate 128 --audio-stream :stereo -o "Lossless Capture Sample-202325.mkv"

To change the file size, which also affects the quality, you can change the --cqp value.
Values that Ive used are in the filenames.

[Selur]

From what I Googled, this would normally result in a file about 1.25 times bigger than 8 bit, which made perfect sense.

No clue what you googled, but it does not make any sense to me.
Assuming we use RGB (= 3 Channels) with
- 8bit = 0-255 values per Channel
- 10bit = 0-1023 values per Channel
so
- 8bit = 256^3 combinations
- 10bit = 1024^3 combinations
....

[Cyber Akuma]

Hmmm, those samples are bigger than I was expecting. Even if I were to use the smallest one, 39MB for 5 seconds is about 19GB total for the whole video. I guess maybe this is something that can't really be reasonably encoded without being monstrously huge. Course, that's also a single 5 second segment.

.... YouTube also somehow also STILL hasn't finished processing the HD version even though it's been about 36 hours now since the upload finished... never seen it take that long.

[jagabo]

I guess maybe this is something that can't really be reasonably encoded without being monstrously huge.

All that motion makes it hard to compress.

[dietboby]

Once NVENC encoding is done, you'll get some info about the encoding of the I, P and B frames.
For this short clip and with the setting of 20:23:25.
This is the result:

frame type IDR   2
frame type I     2,  avgQP  20.00,  total size   0.70 MB
frame type P   106,  avgQP  23.00,  total size  27.89 MB
frame type B   203,  avgQP  24.00,  total size  39.98 MB

You could try to increase the compression of the P and B-frames to get smaller file size.
However, I wouldn't recommend compressing the P-frames too much.
Please note that I went and used the highest level of compression in these examples for the B-frames.

Example with a --cqp 20:23:51 you get

frame type IDR   2
frame type I     2,  avgQP  20.00,  total size   0.70 MB
frame type P   106,  avgQP  23.00,  total size  28.45 MB
frame type B   203,  avgQP  37.00,  total size   7.89 MB

And with a --cqp 20:27:51

frame type IDR   2
frame type I     2,  avgQP  20.00,  total size   0.70 MB
frame type P   106,  avgQP  27.00,  total size  20.59 MB
frame type B   203,  avgQP  39.00,  total size   6.12 MB

I've attached the output files here.
As mentioned the higher the compression the higher the chance that details getting lost.