How do upscale and enhance Full Moton Videos from old video games?

[Guernsey]

I had been looking for ways to upscale and enhance some Full Motion Videos like these for examples to HD. Granted, those YouTube videos have already been converted to HD and have been enhanced but I want to know how to do that for myself? Most of the footage I had show are from very old video game consoles and they tend to have a very low output compared to modern day consoles. Even with NLEs like AVIsynth and FFMPEG, upscaling and enhancing those videos tends to be a very delicate process and you could actually lose some quality. I could be wrong about this but how do I upscale and enhance footage from older gaming consoles or any old piece of animation for Archive.org or YouTube? I am looking for fairly "simple" methods to do this.

[Selur]

Even with NLEs like AVIsynth and FFMPEG,...

Neither FFmpeg nor AviSynth is a NLE.

I am looking for fairly "simple" methods to do this.

Reencoding properly captured content is usually way better than trying go restore content that has tons of compression interations behind it like the content on youtube.
Since when properly capturing you shouldn't have tons of ghosting and compression artifacts.
Restoring multiple compressed content (deblocking, cleaning, color correction, de-ghosting, etc.), from my experience, is by no means archivable by any method I would call 'simple'.

May be you are lucky some tool like Topaz Video Enhance AI works fine.(at least of the cleaning up of the artifacts, no clue whether it can reduce ghosting), anything else that comes to my mind (Vapoursynth, Avisynth, extensive manual filtering in some NLS) is nothing I would call simple.

looking at the first clip you linked to, it's rather easy to clean up a bit:

but I see no way to restore frames without ghosting from that content. (not counting manually redrawing frames or actually frame-by-frame filtering)


=> if you find a 'simple' method let us know (I assume I', not the only one who would be interessted, especially how to get rid of that ghosting,..)

Cu Selur

[Guernsey]

Even with NLEs like AVIsynth and FFMPEG,...

Neither FFmpeg nor AviSynth is a NLE.

I am looking for fairly "simple" methods to do this.

Reencoding properly captured content is usually way better than trying go restore content that has tons of compression interations behind it like the content on youtube.
Since when properly capturing you shouldn't have tons of ghosting and compression artifacts.
Restoring multiple compressed content (deblocking, cleaning, color correction, de-ghosting, etc.), from my experience, is by no means archivable by any method I would call 'simple'.

May be you are lucky some tool like Topaz Video Enhance AI works fine.(at least of the cleaning up of the artifacts, no clue whether it can reduce ghosting), anything else that comes to my mind (Vapoursynth, Avisynth, extensive manual filtering in some NLS) is nothing I would call simple.

looking at the first clip you linked to, it's rather easy to clean up a bit:

but I see no way to restore frames without ghosting from that content. (not counting manually redrawing frames or actually frame-by-frame filtering)


=> if you find a 'simple' method let us know (I assume I', not the only one who would be interessted, especially how to get rid of that ghosting,..)

Cu Selur

What is a good way to do what you had suggested? I was personally hoping for some streamlined methods to upscale and enhance the videos to HD. I will admit that I might be making things more complicated for myself as it isn't as simple as I realized. It seems like very game or piece of media is different so they need to be handled in different ways.

[jagabo]

The first two example videos are poorly upscaled. Just about any program can do that. The third one was better but still not great. A lot depends on what you're starting with. A clean screen cap of a game running in an emulator on your PC has much more potential than a third generation VHS recording poorly captured and mishandled. You should supply a short sample of your video(s) for analysis. Junk from youtube or archive.net usually won't get you good results.

[Guernsey]

This is from the MediaInfo on the sample I provided. I used the Lagarith Lossless codec for the file

General
Complete name                            : *******
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 13.0 MiB
Duration                                 : 45 s 742 ms
Overall bit rate                         : 2 383 kb/s
IsTruncated                              : Yes

Video
ID                                       : 0
Format                                   : Lagarith
Codec ID                                 : LAGS
Duration                                 : 45 s 742 ms
Bit rate                                 : 960 kb/s
Width                                    : 320 pixels
Height                                   : 224 pixels
Display aspect ratio                     : 1.429
Frame rate                               : 59.940 (59940/1000) FPS
Color space                              : RGB
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 0.224
Stream size                              : 5.24 MiB (40%)

Audio
ID                                       : 1
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 45 s 742 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 411.2 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Bit depth                                : 16 bits
Stream size                              : 7.70 MiB (59%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 994  ms (59.59 video frames)

Report created by                        : MediaInfoLib - v21.09
Report created on                        : UTC 2022-01-28 06:05:45

[Selur]

Okay, that source has totally different characteristics than the one I looked at before.
It's pixel art, clean, no ghosting or similar.
So now it maily depends on what 'look' you are aiming for.
The default resizer one could use for such content is the PointResizer:

You could also choose to use some anti-aliasing and normal resizers to get a more cartoon look:


or use some ml based filter for that look:


if you go for ml based filters filters over at the Game Upscale Discord might be interessting.

So you have to make up your mind to you want to try to make such 8bit pixel games look like cartoons or do you want to keep the 'pixel look'?

Cu Selur

[jagabo]

Video examples enlarged 4x with point, nnedi3 (with a little extra):

[pandy]

Perhaps? https://upscale.wiki/wiki/Model_Database

[Guernsey]

Okay, that source has totally different characteristics than the one I looked at before.
It's pixel art, clean, no ghosting or similar.
So now it maily depends on what 'look' you are aiming for.
The default resizer one could use for such content is the PointResizer:

You could also choose to use some anti-aliasing and normal resizers to get a more cartoon look:


or use some ml based filter for that look:


if you go for ml based filters filters over at the Game Upscale Discord might be interessting.

So you have to make up your mind to you want to try to make such 8bit pixel games look like cartoons or do you want to keep the 'pixel look'?

Cu Selur

I was hoping for the latter. I had no idea of the 'cartoony' would also be good. I mostly did some resizing in Virtualdub2 where I did the PointResizing while trying to maintain the aspect ratio and try to soften the pixels with another resize involving Lanzcos. I will have to visit that Discord as I need some tip since some fo the tips I learned about need some reevaluating. Thanks for the tips.

Video examples enlarged 4x with point, nnedi3 (with a little extra):

How did you do those? Can give the script in detail?

Perhaps? https://upscale.wiki/wiki/Model_Database

I have a lot of reading to do. ESGRAN is a new thing for me to learn about.

[hello_hello]

Even with NLEs like AVIsynth and FFMPEG,...

Neither FFmpeg nor AviSynth is a NLE.

But..... Avisynth can access/edit video in a non linear manner.
http://avisynth.nl/index.php/Original_AviSynth_announcement

[jagabo]

Video examples enlarged 4x with point, nnedi3 (with a little extra):

How did you do those? Can give the script in detail?

I used AviSynth and the x264 command line encoder:

point:

AviSource("Sample.avi") 
SelectEvery(8)
PointResize(width*4, height*4)
ConvertToYV12(matrix="rec709")

nnedi3+

 
AviSource("Sample.avi") 
SelectEvery(8)
nnedi3_rpow2(2)
Santiag(4,2)
nnedi3_rpow2(2)
ConvertToYV12(matrix="rec709")
aWarpSharp(depth=10)

The source video was 59.94 fps but each original frame was duplicated about 8 times. I reduced the duplicates with SelectEvery(8). That may not always be the case so you may have to change or remove the SelectEvery() for other sources.

I used the x264 command line encoder in a batch file to encode (drag/drop AVS script onto the batch file):

x264 --preset=slow --crf=20 --keyint=50 --sar=1:1 --colormatrix bt709 --stitchable --output "%~1.mkv" "%~1"

For pixel art video (not the video here) hq2x or hq4x can be good choices:
https://forum.videohelp.com/threads/331048-I-updated-Super-(c)-it-s-giving-HORRID-resu...ts#post2051159

AviSource("PixelArt.avi") 
hq4x()
ConvertToYV12(matrix="rec709")

[Selur]

Cleaning up I stumbled over the file and for the fun of it I used 1x_SSAntiAlias9x to smooth the edges, DPIR for cleanup in Vapoursynth:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
import mvsfunc
# source: 'C:\Users\Selur\Desktop\Sample_8bit.avi'
# current color space: RGB24, bit depth: 8, resolution: 320x224, fps: 59.94, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\Sample_8bit.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Selur/Desktop/Sample_8bit.avi", format="RGB24", cache=0, fpsnum=60000, fpsden=1001, prefer_hw=0)
# making sure frame rate is set to 59.940
clip = core.std.AssumeFPS(clip=clip, fpsnum=60000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/tmp/1x_SSAntiAlias9x.pth"
vsgan.load(model)
vsgan.apply(overlap=16) # 320x224
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=640, h=448, kernel="lanczos", interlaced=False, interlacedd=False)
from vsdpir import DPIR
# adjusting color space from RGB48 to RGBS for vsDPIRDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, range_s="limited")
# deblocking using DPIRDeblock
clip = DPIR(clip=clip, strength=20.000, device_index=0, fp16=True)
# adjusting output color from: RGBS to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited")
# set output frame rate to 59.940fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=60000, fpsden=1001)
# Output
clip.set_output()

Cu Selur