so where's all the Topaz Video Enhance AI discussion?

[FPGA]

...how to implement it with open source softwares only ?

In addition to what Selur mentioned:
Instead of ESRGAN and similar (vsrGAN, realsr, etc...) which are single-image algos - something like TechoGAN could be used (or any other multi-frame-aware inference engine).

[Selur]

I agree using something like TecoGAN (https://github.com/thunil/TecoGAN) might be better suited for video enhancement.

Cu Selur

[forart.it]

...well, seems that someone is (or was ?) working on a Windows-GUI for TECOGAN:

https://github.com/Sanaxen/tecoGAN_app





Hope that inspires !

[Selur]

Yes, nearly a year okd.

[DruidCtba]

...well, seems that someone is (or was ?) working on a Windows-GUI for TECOGAN:

https://github.com/Sanaxen/tecoGAN_app



Hope that inspires !

Very good work dear friend .

Att.

Druid®.

[lollo]

If anybody is interested, I did some more experiment, this time using Topaz VEAI not as an upscaler after an AviSynth restoration flow, but using it directly on the source material, to check its denoise and sharpening features on what is supposed to be the most effective model for live video, Artemis High Quality. The winner, in my case, is still AviSynth

Here the results: https://forum.doom9.org/showthread.php?p=1950219#post1950219 , but the full thread is interesting!

[forart.it]

I recently found this DVD to 1440p upscale:
https://www.youtube.com/watch?v=iUPVczZ2Xq4

The workflow seems interesting:

Software used:
Flowframes (framerate interpolation): https://nmkd.itch.io/flowframes
Topaz Video AI (resolution upscale)
Transition Chooser to clean up scene changes and artifacting: https://www.magentacloud.de/share/lg6gou41fc
Handbrake for DVD ripping
Vegas Pro for audio editing
Photoshop for creating the thumbnail

...how to reproduce it in AviSynth ?

[Selur]

how to reproduce it in AviSynth ?

"Topaz Video AI (resolution upscale)" -> Upscaling alternatives have been discussed here.
Flowframes uses some Frame interpolations that are not available for Avisynth (some are for Vapoursynth), but the results of using FrameRateConverter should not be far away.

[forart.it]

"Topaz Video AI (resolution upscale)" -> Upscaling alternatives have been discussed here.

...of course...

Flowframes uses some Frame interpolations that are not available for Avisynth (some are for Vapoursynth), but the results of using FrameRateConverter should not be far away.

OK...

...and what about Transition Chooser ?

I found this that seems the "official" website: https://christianundco.itch.io/

[Selur]

afaik. that is the tool from the 'Matt Kirby' user from the german Gleitz Forum (https://gleitz.info/forum/index.php?thread/48239-zwischenbildberechnung-mit-ki-hat-jem...560#post463560).
It's uses FlowFrame to tweak the interpolation/scene change detection:

The Transitionchooser FLOWFRAMES-EDITION 8.0:
It's a tool for users that want to interpolate (2x, 4x, 8x) long movies with FlowFrames.
Interpolation is cool but on scene changes it produces ugly interpolated frames.
That disturb while watching the video. FlowFrames has already a scene detection for this problem.
But no scene detection is perfect and the users have to edit transitions manually.
With TC you can change transitions or you can add new ones manually if the automatic detection has missed some.
Some users also add new transitions to cover "artifact frames" with blend-frames, for example.
Now (with ver. 8.0) it is possible to add a alternative interpolfolder from another AI.
So you can choose a 4th frame, an orange interpolframe.


Transitionchooser needs to install avisynth+ 32bit !!!

Workflow:
- disable autoencode in FlowFrames settings
- enable scene detection in FF settings
- run FlowFrames in step by step Mode
- run the first steps until interpolation step
- if interpolation step was done leave FF open and open the "Transitionchooser"
- open the FlowFrames-ini in the temp-folder that FF has created for your video ("frames-2x.ini" or "frames-4x.ini")

Transitionchooser reads now "ini" in "temp-folder" and lists all transitions made by FlowFrames scene detection.
Then push button "Make Blendframes" and wait, for an entire movie it can last some minutes

And now you can choose which pic should be at a transition (scene change)

- Do you wanna have a blend frame(fading frame(s) for smooth transition) ?!
- Do you want a double frame (the last frame of the previous scene will be repeated = doubled) ?!
- Or do you like the morphed interpolated pic for the case that the automatic scene detection has made a double frame where no transition really is.
and "red" interpolated pic will be the same as there is no scene change, choose this if FlowFrames has detected a false positive scene change
- or do you want to hide artifacts with alternative "orange" interpolframe, when you have added an alternative folder

In the preview you can watch each transition (scene change) previous frame, transition frame and nextframe will be shown
with pushing "space"-key you can change each frame to another sort
The workflow will be: "go with cursor down through the list and look at the two frames
"previous" and "next" if there is a ugly frame between change it and go further, so you can
edit quickly a entire movie

If you have set your choice through the whole list you just must push "modify-ini button"
(optional you can save your edited list, if you want to edit it later again, and you can
output the pngs that have changed if you use an external encoder)
or you can output all frames as an entire imagesequence

Now you can close Transitionchooser!

Go back to the open FlowFrames window and run there the last step ("export step")

So could you do something similar with FrameRateConverter and some scripting in Avisynth: Yes, but you would need to understand Avisynth, also it's not really related to the topic of this thread.


Cu Selur

[forart.it]

(Fresh find) Real-ESRGAN aims at developing Practical Algorithms for General Image Restoration.





Git: https://github.com/xinntao/Real-ESRGAN

Colab-demo: https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo

Now available in MixLab too: https://colab.research.google.com/github/shirooo39/MiXLab/blob/master/MiXLab.ipynb#scr...o=Jbw2QIUB6JKR

Enjoy !


EDIT: just tested the MixLab version

ORIGINAL


4X OUTPUT

[Attachment 61044 - Click to enlarge]

[rrats]

(Fresh find) Real-ESRGAN aims at developing Practical Algorithms for General Image Restoration.

esrgan looks like sharpened waifu2x

[Selur]

btw. there's Vaporusynth version https://github.com/HolyWu/vs-realesrgan/ of it which works fine.

[rrats]

I recently found this DVD to 1440p upscale:
https://www.youtube.com/watch?v=iUPVczZ2Xq4

The workflow seems interesting:

Software used:
Flowframes (framerate interpolation): https://nmkd.itch.io/flowframes
Topaz Video AI (resolution upscale)
Transition Chooser to clean up scene changes and artifacting: https://www.magentacloud.de/share/lg6gou41fc
Handbrake for DVD ripping
Vegas Pro for audio editing
Photoshop for creating the thumbnail

...how to reproduce it in AviSynth ?

thats idiotic. no normal person would interpolate frames

[forart.it]

Well then seems that Denis Malimonov putted many interesting resources togheter into a colab notebook:

Ensemble of Neural Tools for Animations Restoration

This ensemble includes:
- DeOldify
- EDVR
- VFIASC
- ESRGAN

In addition, there are:
- DeepRemaster
- Fast-SRGAN
- VFI-CFT
- DeOldify Stable (new)
- 3D Ken Burns
- DDSP (new)

I didn't tested it yet, check it out:
https://colab.research.google.com/github/tg-bomze/ENTAR/blob/master/ENTAR_Eng.ipynb

[rrats]

Well then seems that Denis Malimonov putted many interesting resources togheter into a colab notebook:

Ensemble of Neural Tools for Animations Restoration

This ensemble includes:
- DeOldify
- EDVR
- VFIASC
- ESRGAN

In addition, there are:
- DeepRemaster
- Fast-SRGAN
- VFI-CFT
- DeOldify Stable (new)
- 3D Ken Burns
- DDSP (new)

I didn't tested it yet, check it out:
https://colab.research.google.com/github/tg-bomze/ENTAR/blob/master/ENTAR_Eng.ipynb

Interesting

[rrats]

I have finally managed to set up vsgan

[taigi]

So, what is best for upscaling in summa summarum without using Topaz in 2021?

[Selur]

Summa summarum: depends on your source and available hardware (especially of the ml based stuff need modern NVIDIA cards with lots of RAM).

[taigi]

@forart.it how to batch Real-ESRGAN?

[poisondeathray]

@forart.it how to batch Real-ESRGAN?

You can do a windows batch with commandline using wildcard *.* - official repository comes with a compiled binary
https://github.com/xinntao/Real-ESRGAN/releases

eg. something like this

for %i in (*.*) do "PATH\realesrgan-ncnn-vulkan.exe" -i "PATH\%i" -o "PATH\%~ni_output.png"

Or use vapoursynth vs-realesrgan to batch, or some GUI's have Real-ESRGAN vulkan too (e.g.
waifu2x-extension-gui)

https://github.com/AaronFeng753/Waifu2x-Extension-GUI

[JoelHruska]

"So, what is best for upscaling in summa summarum without using Topaz in 2021?"

I have yet to find anything remotely as good as Topaz when performance is tossed into the equation. I continue to experiment, especially since TVEAI has problems in certain footage. When used on older DVD-encoded content, however, it can be made to perform well -- provided you're willing to do a lot of work with AviSynth first.

The following comparisons are from my own work on Star Trek: Deep Space Nine. They were created using AviSynth, Topaz Video Enhance AI, and DaVinci Resolve Studio 17. I have included several comparisons where Topaz still struggles a bit, to illustrate both the ups and downs. This comparison episode is from Season 3 -- Seasons 1 - 3 are much lower quality than S4 - S7 and are harder to remaster. This is a tough workload for Topaz compared to the later seasons of the show.

The OG DVD is on the left in every case. The OG DVD has been de-telecined without the application of any other filters.

https://imgsli.com/NzcyNTU

https://imgsli.com/NzcyMzg

https://imgsli.com/NzcyMzk

https://imgsli.com/NzcyNDA DS9 S1 - S3 are heavily rainbowed. You can see it in Kira's hair, in the OG DVD. Processing with AviSynth removed the rainbowing. (TVEAI can sometimes reduce rainbowing slightly, but it's a side-effect of the upscaling process, not a feature and it's a small reduction in any case). I trade a tiny bit of shoulder and facial detail here, but getting that back is as simple as adjusting my compositing parameters. Not a problem. Because I'm using different models, I can weight the final output towards sharpness or towards smoothness.

https://imgsli.com/NzcyNDI

https://imgsli.com/NzcyNDE Topaz struggles with this one. Eyes come out rough. The white blowout in the upscaled version is my doing, not Topaz. I'm experimenting with various methods of compositing in DaVinci and that happened to be a side-effect of the process.

https://imgsli.com/NzcyNDY This is one of the best shots of how effectively Topaz can improve content.

https://imgsli.com/NzcyNDc This is an older version of the comparison immediately above. I'm showing this comparison because it has no color grading and offers a better apples-to-apples comparison against the DVD. It's older, so there's also a small quality hit -- check Jadzia's com badge for the difference -- but it's still a good comparison.

I'm not going to pretend this is simple. The output I've uploaded is a blended mixture of no fewer than *six* different Topaz encodes and three different AviSynth encodes. It's taken me a long time to figure out which AviSynth script options and Topaz models would work best together. I can tweak various options, like how much noise is in the final output, by which outputs I select to blend.

https://imgsli.com/NzcyNDk

This shot shows a comparison between where I was in mid-August, when I created my first upscale of the episode and where I am today, two months later. This is upscale versus upscale, not upscale-versus-OG DVD.

I continue to use Topaz because I have found no alternate model capable of delivering these results at a speed of 0.07s - 0.10s per frame. But I've also seen no model that was capable of matching this quality improvement, period. I am utterly agnostic as far as software is concerned and am willing to use / learn any product that can deliver superior results. I just haven't found one yet.

[rrats]

So you have enough price to buy topaz, but not to activate bloody windows @JoelHruska

[Selur]

@JoelHruska: have you tried how RealESRGAN and BasicVSR++ handles your content?

[JoelHruska]

So you have enough price to buy topaz, but not to activate bloody windows @JoelHruska

I do this work on a mixture of testbeds and my own system. I don't activate testbeds because their OS installations don't necessarily live very long. That means, yes, there's some stupid watermarking in some of my work.

[JoelHruska]

@JoelHruska: have you tried how RealESRGAN and BasicVSR++ handles your content?

Selur,

I have tried all of the models you were kind enough to share a few months back and a few I found since. I do not believe I have specifically tested RealESRGAN or BasicVSR++. I would be quite interested in doing so.

[poisondeathray]

RE:BasicVSR++ : The default models tend not to be good for typical DVD sources , including Star Trek or similar genres - they were not trained on anything like that and I haven't seen any other publicly distributed models yet.

BasicVSR++ just can't handle typical DVD noise and softness; and if you prefilter /denoise it tends to cancel much of the benefit or causes other problems. I tested scenes from about a dozen DVD's of variety of genres. The current models are not trained for DVD's

2 pluses:

1) It's the one of the very few GANs that does well on text/letters/numbers... provided the source was clean .
2) The temporal stability is more like tecogan or one of the MC VEIA models like Artemis-HQ.

(This is an apng it should animate in most browsers; if not click on it or open in new tab)


But limited real world usefulness - because you need a clean source. Great on academic or synthetic tests where you downscale a source to upscale for testing, but that's about it unfortunately

If you have a messed up upscaled text, maybe the console in the background or something , it might be worth looking at for a specific scene

Or if anyway figures out a way to improve it with more typical scenarios, please post

[Selur]

@JoelHruska: those are not models for VSGAN
-> I'll send you a link to my current Hybrid dev version and my current torch addon (renamed the VSGAN addon since it includes a bunch of other torchpy based filters like RealESRGAN).
(BasicVSR++ might only be interessting for short scenes)

https://imgsli.com/NzcyNDY This is one of the best shots of how effectively Topaz can improve content.

did you share a sample clip where this was included?

Cu Selur

[lordsmurf]

https://imgsli.com/NzcyNDY This is one of the best shots of how effectively Topaz can improve content.

did you share a sample clip where this was included?

That's a bogus before/after.
The color changes have an affect on perceived sharpness. So it's worthless.

[forart.it]

The current models are not trained for DVD's

Well, my main goal is to upscale DVD (720p) sources at a - reasonable - 2X resolution...

Do you think is possible to train algos with well-known movies released at 1080p ? (example: Criterion BDs)

EDIT
A (maybe) interesting resource about BasicVSR++

- BasicVSR++ function for VapourSynth