How do you avoid artifacts from 3D game captures?

[Guernsey]

Whether it is emulated game capture or direct game capture, I want to upscale them to HD but I do not want any artifacts. I know YouTube will re-encode your captures anyway but I want the videos to be presentable to watch. I also do not want the videos to go past a certain point but Virtualdub may not be that editor as I need some codecs for a direct stream copy. I was once told that I need to upgrade to H265 but I deal with game captures from fourth to sixth generation game consoles.

I am not that familiar with H265 as I am with H264 and I mostly used the x264 8 bit codec. Also, some video games used animation or real life footage so I need to keep that in mind too. What can I do to avoid artifacts? Do I need to use lossless codecs? Do I need to use a different video editor altogether? How do you avoid artifacts from game captures?

[Selur]

What artifacts you are talking about?
Assuming your source is pristine (no transmission or initial artifacts) and properly flagged:
a. There are artifacts caused by compression. To avoid these, a more lossless encoding is needed.
b. There are artifacts caused by filtering (or improper filtering). To avoid these, you have to avoid filtering or keep them in mind while filtering and maybe remove them.

Switching to H265 from H264 only helps to lessen compression artifacts, since:
a. H265 has better compression/quality ratio, so with the same size less compression artifacts appear.
b. H265s main profile supports 10bit, thus encoding in 10bit will avoid causing banding while not hindering playback compatibility.

The only way to make sure to avoid any additional artifacts on a pristine (and properly flagged) is to keep the source as it is or:
a. not apply any filtering and
b. use lossless compression which supports the needed flags.

=> If you need/want further help, try to make it more clear what artifacts you want to avoid.

Cu Selur

[jagabo]

You want to capture at native frame size with a lossless codec. Then you worry about editing, upscaling, and encoding. h.264 and h.265 are for your final delivery to youtube, not for capture.

Do you have some samples you can upload here?

[Guernsey]

Here is the media information of both files and I did not upscale them yet.:

General
Complete name             : Sample.avi
Format                    : AVI
Format/Info               : Audio Video Interleave
Format settings           : BitmapInfoHeader / PcmWaveformat
File size                 : 279 MiB
Duration                  : 1 min 43 s
Overall bit rate mode     : Variable
Overall bit rate          : 22.6 Mb/s
Frame rate                : 59.940 FPS
Writing library           : VirtualDub2 build 44282/release

Video
ID                        : 0
Format                    : FFV1
Format version            : Version 1
Codec ID                  : FFV1
Duration                  : 1 min 43 s
Bit rate mode             : Variable
Bit rate                  : 21.2 Mb/s
Width                     : 352 pixels
Height                    : 240 pixels
Display aspect ratio      : 3:2
Frame rate                : 59.940 (59940/1000) FPS
Color space               : RGB
Bit depth                 : 8 bits
Compression mode          : Lossless
Bits/(Pixel*Frame)        : 4.178
Stream size               : 261 MiB (94%)
coder_type                : Range Coder

Audio
ID                        : 1
Format                    : PCM
Format settings           : Little
Codec ID                  : 1
Duration                  : 1 min 43 s
Bit rate mode             : Constant
Bit rate                  : 1 411.2 kb/s
Channel(s)                : 2 channels
Sampling rate             : 44.1 kHz
Bit depth                 : 16 bits
Stream size               : 17.5 MiB (6%)

General
Complete name             : Sample 2.avi
Format                    : AVI
Format/Info               : Audio Video Interleave
Format settings           : BitmapInfoHeader / PcmWaveformat
File size                 : 92.0 MiB
Duration                  : 1 min 32 s
Overall bit rate mode     : Variable
Overall bit rate          : 8 383 kb/s
Frame rate                : 59.940 FPS
Writing library           : VirtualDub2 build 44282/release

Video
ID                        : 0
Format                    : FFV1
Format version            : Version 1
Codec ID                  : FFV1
Duration                  : 1 min 32 s
Bit rate mode             : Variable
Bit rate                  : 6 949 kb/s
Width                     : 320 pixels
Height                    : 224 pixels
Display aspect ratio      : 1.429
Frame rate                : 59.940 (59940/1000) FPS
Color space               : RGB
Bit depth                 : 8 bits
Compression mode          : Lossless
Bits/(Pixel*Frame)        : 1.617
Stream size               : 76.3 MiB (83%)
coder_type                : Range Coder

Audio
ID                        : 1
Format                    : PCM
Format settings           : Little
Codec ID                  : 1
Duration                  : 1 min 32 s
Bit rate mode             : Constant
Bit rate                  : 1 411.2 kb/s
Channel(s)                : 2 channels
Sampling rate             : 44.1 kHz
Bit depth                 : 16 bits
Stream size               : 15.5 MiB (17%)

I used the FFV1 Lossless codec from the FFMPEG writer of an emulator to dump the video. I also used my Virtualdub2 to cut video clips to prevent it from becoming too big to upload. I did this on Chromebook although I should have used Windows as my "Linux" (Linux Beta on Chrome OS) does not have all the codecs.

Edit:

What artifacts you are talking about?

I am trying to prevent blurring, color bleeding and possible blocking. I do not want to see any distortions such as when an object becomes to blurry to see

[jagabo]

What kind of scaling are you looking for? Like these? And how much scaling? Those are 4x.

[Selur]

I agree with jagabo, Point resize would be the most accurate for pixel art: https://imgsli.com/MjgzMzMz
Stuff like hq4x and similar filters will try to 'modernize' the look,...

[jagabo]

Also note:

Point resize works best when scaling by an integer amount. For example, 320x224 to 640x448 or 1280x896.

I believe Youtube will convert everything to YUV 4:2:0 chroma subsampling. Which means that you will always get some blurring of colored edges there.

[Guernsey]

What kind of scaling are you looking for? Like these? And how much scaling? Those are 4x.

I was hoping for something like 2xSal for pixel art but for the most part, I was looking for something that can work with emulated game captures. So do I just use Point Resize for pixel art but what about for 3D models, animation or live action footage? Or does that fall into the same purview as pizel art? Some of these games were mostly on older consoles as opposed to something more modern such as the current PlayStation 5 or the PC.

[Selur]

The more natural the content, the more you usually don't want to have strict resizing.
So in general, I would recommend trying NNEDI3, it also depends on personal preferences.

[Guernsey]

Do you mind giving me the script? I could use it with MeGUI.