Bandicam - differences between NVENC H264 and CPU H264?

[CursedLemon]

Hey guys. I've been routinely capturing PC video game videos using Bandicam, which gives two options for H264 capture, one being NVENC and the other being QuickSync - or I assume it's QuickSync, I have an i7 6700K and it just says "H264 (CPU)".

I've been capturing in an .avi container, 60 FPS and 100% quality setting. For a while I've been using the NVENC option, if for literally no other reason than it's cool to finally have a video program that will actually use my GTX 960 to some extent of its power. :B

Problem here is that whenever I try to utilize the videos I've captured, namely either in VirtualDub or in Sony Vegas, I have all kinds of problems handling these capture files. VirtualDub will stutter and freeze when trying to play them back, and if I try to use direct stream copy to crop the video, the resulting file will be useless. Similarly, trying to use these captures in Vegas will result in visual artifacts (even after render), improperly loading audio tracks, and problematic playback. I also think that these files were giving MediaCoder a hard time when trying to convert them down to smaller size.

On a whim, I happened to change the H264 capture setting in Bandicam to use the CPU instead...and all of these problems have seemingly vanished.

My assumption was that the video files generated by NVENC or QuickSync would be the same thing, just utilizing different technologies. Apparently this isn't so, so I was wondering if someone could elaborate on what the technical differences are between these methods, and perhaps why the NVENC files are so troublesome?

Here are the MediaInfo exports for one of each file.

NVENC:

General
Complete name                            : C:\Users\Cursed Lemon\Desktop\MassEffect3 2016-08-21 13-39-57-618.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Format profile                           : OpenDML
File size                                : 1.84 GiB
Duration                                 : 5 min
Overall bit rate                         : 44.0 Mb/s
Writing application                      : BandiAviMuxer 1.0

Video
ID                                       : 0
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4.2
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 1 frame
Codec ID                                 : H264
Duration                                 : 5 min
Bit rate                                 : 43.8 Mb/s
Width                                    : 1 440 pixels
Height                                   : 900 pixels
Display aspect ratio                     : 16:10
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.563
Stream size                              : 1.83 GiB (100%)

Audio
ID                                       : 1
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Codec ID                                 : 50
Duration                                 : 5 min
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Compression mode                         : Lossy
Stream size                              : 8.22 MiB (0%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 24  ms (1.44 video frame)

QuickSync:

General
Complete name                            : C:\Users\Cursed Lemon\Desktop\MassEffect3 2016-08-21 14-05-50-644.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 12.4 MiB
Duration                                 : 8 s 736 ms
Overall bit rate                         : 11.9 Mb/s
Writing application                      : BandiAviMuxer 1.0

Video
ID                                       : 0
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Baseline@L4.2
Format settings, CABAC                   : No
Format settings, ReFrames                : 1 frame
Codec ID                                 : H264
Duration                                 : 8 s 733 ms
Bit rate                                 : 11.7 Mb/s
Width                                    : 1 440 pixels
Height                                   : 900 pixels
Display aspect ratio                     : 16:10
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.151
Stream size                              : 12.2 MiB (98%)

Audio
ID                                       : 1
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Codec ID                                 : 50
Duration                                 : 8 s 736 ms
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Compression mode                         : Lossy
Stream size                              : 205 KiB (2%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 24  ms (1.44 video frame)

[LigH.de]

As already mentioned several times, the H.264 video format is too complex to be contained reliably in an AVI container, because the AVI container does not expose all the attributes about each H.264 video frame a video editor would need to know, and does not even support B frames (references to not yet displayed frames) in general. Video editors focused only on features known to be supported by the AVI container will miss important details in H.264 video streams and mess up the video, the saved result may not play without decoding errors anymore. And free but still powerful video editors supporting a wide range of input formats are sparse... VirtualDub may not be suitable anymore (maybe with ffmpeg input plugin), Avidemux may not be as intuitive for every user, and the rest?!

H.264 may be a quite efficiently compressing video format; but neither should it be stored in the AVI container (instead rather in MP4, if available, or alternatively in MKV or M2TS, or maybe MOV as last chance), nor should it be used for intermediate files which are meant to be edited further, except with very restricted structure details which would reduce the compression efficiency but improve the editability (these two goals are mutually exclusive). If you want to cut videos which were stored in H.264 format at arbitrary positions or even change their frame content (e.g. apply filters or just crop their dimensions), expect that you may have to recompress the whole video, which will be quite time consuming and quality reducing.

The choice of the encoder will have an impact on the quality. H.264 encoders using a hardware encoder in the GPU are notorious for producing only video with quite limited complexity, therefore not quite optimal quality per bitrate. If you have a CPU with many cores, you should probably prefer a CPU encoder which may be either even faster than the GPU (depending on the chipset) or a lot better and still quite fast. But I would suggest to use the H.264 format only for the final output, not for intermediate clips.

[CursedLemon]

The sole reason I like to capture in .avi format is so that I can do "direct stream copy" video cropping in VirtualDub (if there's any other program or file format that can perform that kind of operation, I'm all ears).

Should I be capturing my video in YV12 format, then?

[LigH.de]

"Cropping" means reducing the width or height of the video area; this is impossible in "Direct Stream Copy" video mode because it would alter the video content, thus require a complete re-encoding.

You probably mean "cutting" (removing scenes between two frame numbers in the timeline). This can only be done losslessly when you use a video format where all video frames are encoded independently (e.g. MJPG or DV), or you restrict your cut points to keyframes only. But H.264 uses a lot of dependencies between the content of frames. Most frames after a cut point won't be decodable without errors if they don't follow an IDR frame. Unfortunately, the AVI container only knows "keyframes" and will not know the difference between IDR frames and "inserted I frames" (which may be crossed by references). If you cut at an "inserted I frame", there will still be decoding errors afterwards.

Capturing PC games gives you even worse issues: You will quite certainly lose chrominance details. On one hand, RGB video as used onscreen may use full range colors, but YUV video as used in efficiently compressed video formats usually has a limited range of luminance and chrominance (except you know how to avoid that); and on top, it uses most often the chroma subsampling 4:2:0, which means that the chrominance has a lower resolution in both dimensions, means, colors may "bleed" into the neighborhood, which may not be very obvious for natural scenes, but computer video has extremely sharp edges...

I guess you will find several more existing threads about such issues when you look for "FRAPS".

[poisondeathray]

You can't crop with stream copy

Did you mean "trimming ?" ie. cutting sections by time ? "cropping" usually refers to cutting away the frame borders (eg. left, top, right, bottom) so you're left with a smaller frame dimension. Trimming with stream copy can usually only be done on keyframes with free software, you can use avidemux, xmediarecode . Frame accuracy requires partial re-encoding of split GOP segments, common tools would be videoredo, solveigmm video splitter, tmpg smart renderer

When using bandicam, MP4 will be more reliable in vegas and every other program when using AVC

EDIT: posted too slow

[CursedLemon]

Yeah, "trimming" is what I actually meant, haha.

So is there really a terribly decent solution for creating PC video game capture files that are legitimately editable? There's no way I can use RGB as a capture medium because I just simply don't have the disk space.

[LigH.de]

You will probably lose some chrominance anyway when compressing to H.264 in the end (and certainly when you upload to YouTube, because YouTube will recompress everything once more); so you can use e.g. the MJPEG VfW codec as provided by ffvfw, part of the already a bit obsolete ffdshow. Also there are good lossless codecs like the Ut Codec suite or FFV1 (in ffvfw), but you will have to recompress if you don't cut at keyframes.

I don't know under which circumstances VirtualDub may support "smart rendering" (only recode a cut range between keyframes, copy the unaltered rest).

But the smartest solution may be: cutting your videos only logically, e.g. in an AviSynth script. In this case you can record in any intermediate format you like, don't need to save a second cut edition as another huge video file, just the script text (the video will be sent to the encoder as the remaining scenes), but you need to learn video scripting.

[El Heggunte]

So is there really a terribly decent solution for creating PC video game capture files that are legitimately editable? There's no way I can use RGB as a capture medium because I just simply don't have the disk space.

RGB and YUV (actually YCbCr) are color spaces, not compression types

If Bandicam doesn't let you choose and configure different VfW codecs, then you'd better switch to a different application.
Video capture works best with lossless compression or (at least) with intra-frame-only lossy compression,
neither of which is supported by Bandicam

[CursedLemon]

You will probably lose some chrominance anyway when compressing to H.264 in the end (and certainly when you upload to YouTube, because YouTube will recompress everything once more); so you can use e.g. the MJPEG VfW codec as provided by ffvfw, part of the already a bit obsolete ffdshow. Also there are good lossless codecs like the Ut Codec suite or FFV1 (in ffvfw), but you will have to recompress if you don't cut at keyframes.

I don't know under which circumstances VirtualDub may support "smart rendering" (only recode a cut range between keyframes, copy the unaltered rest).

But the smartest solution may be: cutting your videos only logically, e.g. in an AviSynth script. In this case you can record in any intermediate format you like, don't need to save a second cut edition (the video will be sent to the encoder as the remaining scenes), but you need to learn video scripting.

I really should just buck up and learn AviSynth, I mean I work IT for a living and I spend a lot of time in Linux so it sure seems like the logical route, eh?

So is there really a terribly decent solution for creating PC video game capture files that are legitimately editable? There's no way I can use RGB as a capture medium because I just simply don't have the disk space.

RGB and YUV (actually YCbCr) are color spaces, not compression types

If Bandicam doesn't let you choose and configure different VfW codecs, then you'd better switch to a different application.
Video capture works best with lossless compression or (at least) with intra-frame-only lossy compression,
neither of which is supported by Bandicam

I guess by RGB I just meant fully uncompressed in RGB24.

Bandicam does let you use external codecs, but it seems to take a performance hit when using them, as if it's not optimized for them. I can, for example, capture using the Lagarith codec.

[El Heggunte]

Yes, Lagarith is lossless, but I'm afraid it's not very-suitable for real-time video captures
It would be great IF Bandicam allowed the user to choose faster lossless codecs, such as HuffYUV, CamStudio, MagicYUV or UTVideo.

Anyway: regarding low disk space... you really should get another HDD, even an external one,
and use FRAPS (for example) instead of Bandicam. That's what I did, when I had a videogame channel on YouTube.