Upscaling after sRestore leads in huge artefacts & sRestore with Vapoursynt

[Platos]

I have 2 Problems:

1. When i use Avisynth QTGMC (very slow setting) + Bob and sRestore (default setting and no other filter), it leads in very strange artefacts, when i upscale then with RealCUGAN. If i use only Avisynth QTGMC (without Bob) this does not happen. And before upscaling it looks completly fine, so it has not a single of these kind of artefacts. Problem is, i have to delete this blended frames before Upscaling, otherwise the Upscaling does not work that good as it could.

Here the video with the artefacts: https://uploadnow.io/f/TQZjF82

2. When i use Vapoursynth QTGMC + Bob + sRestore i can't reduce frame blending. It does just not remove the blended frames (not a single one). It happens nothing actually. Why ?

I use Hybrid for all of that, if it is important.

Edit: Here the two original video i used ("kurz 2" is from question 1, the other for question 2): https://uploadnow.io/de/share?utm_source=6w435XS

[poisondeathray]

I have 2 Problems:

1. When i use Avisynth QTGMC (very slow setting) + Bob and sRestore (default setting and no other filter), it leads in very strange artefacts, when i upscale then with RealCUGAN. If i use only Avisynth QTGMC (without Bob) this does not happen. And before upscaling it looks completly fine, so it has not a single of these kind of artefacts. Problem is, i have to delete this blended frames before Upscaling, otherwise the Upscaling does not work that good as it could.

Here the video with the artefacts: https://uploadnow.io/f/TQZjF82

2. When i use Vapoursynth QTGMC + Bob + sRestore i can't reduce frame blending. It does just not remove the blended frames (not a single one). It happens nothing actually. Why ?

I use Hybrid for all of that, if it is important.

Edit: Here the two original video i used ("kurz 2" is from question 1, the other for question 2): https://uploadnow.io/de/share?utm_source=6w435XS

1) Not sure what those artifacts are. I can't get the .dll to load in vapoursynth for RealCUGAN to test it. The avisynth version works ok. Not sure if it's possible to use the avs version in hybrid

2) There was a change in vapoursynth API4 that "broke" the linear cache required for srestore . It's being looked at by developers . API3 works ok, not sure how you'd switch versions in hybrid, selur will tell you if it's possible
https://forum.doom9.org/showthread.php?t=184852

[poisondeathray]

Could it be you're using weird tile settings ? Some memory issue ?

I'm having trouble loading the .dll from this version to test it
https://github.com/Kiyamou/VapourSynth-RealCUGAN-ncnn-Vulkan

But vs-mlrt works ok. The avs version is actually a partial port of this one
https://github.com/AmusementClub/vs-mlrt

Not sure which one is bundled with hybrid

[Platos]

Thanks for your answer.

Maybe i was to unprecise, sorry. The upscaling with RealCUGAN i do with Waifu2x-Extension-GUI (https://github.com/AaronFeng753/Waifu2x-Extension-GUI)

The 2 problems i have was in Hybrid. So the sRestore i do with Hybrid. With vapoursynth in Hybrid it does not work (how i said in Question 2), with avisynth it does. But the uscaling i do with Waifu2x-Extension-GUI because it's 2.3x times faster than hybrid with RealCUGAN. Sorry, i should have be more precise.

But it's not the Waifu2x-Extension-GUI. I use this software over a year now and i had never such problems and it also does not appear, if i do not use sRestore with Hybrid. But i'm not sure if its a Hybrid-problem or a sRestore-problem.

[Edit: I tried it now to with Hybrid and RealCUGAN Vapoursynth. It produces the exactly same artefacts]

But vs-mlrt works ok. The avs version is actually a partial port of this one
https://github.com/AmusementClub/vs-mlrt

I don't know, what vs-mlrt is. So in Hybrid i have to use Vapoursynth to use RealCUGAN. There is no option for Avisynth.

2) There was a change in vapoursynth API4 that "broke" the linear cache required for srestore . It's being looked at by developers . API3 works ok, not sure how you'd switch versions in hybrid, selur will tell you if it's possible
https://forum.doom9.org/showthread.php?t=184852

Ah, interesting. I did not saw there is already a thread. Funny thing is, the scene he is talking about was from my question some days ago. But i did use avisynth, so i did not saw the problem.

Actually it's not a huge problem. I just can use Avisynth. But for sure it would be nice, when it will work in an future update. Because it's 5x faster on my setup (50 vs 10fps).

So but then the Question 2 is solved basically. Question 1 was more important for me anyway. I dont want to stress selurs balls with a bug, he already knows He helped me enough already

I have time

[Selur]

Vapoursynth + sRestore: like poisondeathray mentioned, looking into it.

not sure how you'd switch versions in hybrid, selur will tell you if it's possible

not supported

for RealCuGAN: have you tried a different model, especially 'pro' seems to cause issues in some sources.

[Platos]

Vapoursynth + sRestore: like poisondeathray mentioned, looking into it.

not sure how you'd switch versions in hybrid, selur will tell you if it's possible

not supported

for RealCuGAN: have you tried a different model, especially 'pro' seems to cause issues in some sources.

I will try, but i use always Model se, so i can try "non se" and "pro" (i dont know what model "non-se" should be).

But the thing is, it does not happen, when i do not use sRestore. Then there is no artefact. So that's why i think, it's sRestore.

If i can test sth more, tell me. I test now with other models.

I made an Edit in the last answer. The same artefacts (like really exactly same position and time) happens too, when i use hybrid RealCUGAN.

Edit: Could it be possible, that the downsizer (lanczos) or resizer (10 bicubic spline) does make this happen ?

[poisondeathray]

vsmlrt is another implementation . You can run CUGAN in it

To rule out if RealCUGAN is the culprit, or maybe some other threading issue. Run avs srestore in hybrid, save a lossless intermediate. Load the intermediate then run RealCUGAN in hybrid

I get no artifacts using vs-mlrt CUGAN using the pro model ( pro-conservative-up2x)

[Platos]

What is culprit?

How can i make a lossless intermediate or what is an intermediate ?

I did QTGMC + Bob and sRestore Avysynth, saved it (Nvenc 265, quality, 10 bit, 2 full passes, quarter pixel, 1.5mbit/S variable bitrate, main10.) and then i load it in hybrid again (this new video) and it produces those artifacts with model Se and non-Se. Model Pro crashes in Hybrid. I tried it 3 times

In Waifu2x-Extension-Gui it produces the same artefacts with model se and model Pro. So both software same problem. Tried all models available there.

Can you tell me, where/how i can use this vsmlrt ? i have like zero knowledge of commands with videoencoding.So is there a gui ?

[Selur]

Had a look at the artifacts: those look like a memory issues. GPU/System too warm?

[poisondeathray]

Yes, I'm thinking local system problems if Waifu2x-Extension-Gui also has the problem

vs-mlrt is another implementation for various machine learning projects in vapoursynth . I don't know of any GUI's unless hybrid has it. There are instructions and examples in the documenatation
https://github.com/AmusementClub/vs-mlrt

[Selur]

No Hybrid doesn't support vs-mlrt. ( I looked into it, but couldn't motivate myself to create another huge addon for Hybrid)
I used:

# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Fast", TFF=True, opencl=True) # new fps: 50
# Making sure content is preceived as frame based
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
# adjusting frame count and rate with sRestore
clip = havsfunc.srestore(source=clip, frate=36.9880)
clip = core.std.AssumeFPS(clip=clip, fpsnum=9247, fpsden=250)
clip = havsfunc.srestore(source=clip, frate=23.9760)
clip = core.std.CropRel(clip=clip, left=12, right=12, top=8, bottom=2)# cropping to 696x566
clip = core.std.AddBorders(clip=clip, left=0, right=0, top=0, bottom=2) # add borders to archive mod 4 (vsRealCUGAN) - 696x568
# adjusting color space from YUV420P8 to RGBS for vsRealCUGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# resizing using RealCUGAN
clip = core.rcnv.RealCUGAN(clip=clip, noise=0, model=2, syncgap=2) # 1392x1136
# resizing 1392x1136 to 1280x1042
clip = core.std.CropRel(clip=clip, left=0, right=0, top=0, bottom=4) # removing borders (vsRealCUGAN) -  1392x1132
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1280, h=1042, kernel="lanczos", interlaced=False, interlacedd=False)

on both clips. (notice: I used an ugly workaround to 'fix' sRestore, by calling it twice)
So it's not an issue with the RealCUGAN an the source clips.

Cu Selur

[Platos]

Your test refer to my question why there are artefacts, right? Or was this a test about the Vapoursynth-problem? Did you use hybrid on this test? But i think then it is no RealCUGAN problem and it can also not be a Hybrid-problem, because it happens on both software. So it has to be a sRestore problem, right ?

Why did you not use avisynth-sRestore in normal way (in hybrid)? Does avisynth also have a bug like the Vapoursynth? Or why do you use it 2 times ? Because that brings more variable in the case.

And i do not have temperatur problems. Under 60° for CPU and GPU. And it's basically impossible, that it is a system-problem. I can use RealCugan with the original source Video and it has no artefacts while it has artefact, when i before edit the source-video with sRestore (avisynth) in Hybrid. So i have no problem, when i dont use sRestore.

And about vs mlrt: This is above my knowledge without a gui, sry.

And sorry for late answer. I was not there.

[Selur]

The problem with sRestore and Vapoursynth atm. is that in the last quarter of the last year Vapoursynth updated to a new API.
This API broke some stuff that causes issues with sRestore, through which it will not properly decide which frames to reduce.
Only workaround I found so far is to call sRestore 2 times.

The artifacts Naruto DVD Folge 10_kurz2_avi sRestore + deinterlaced_waifu2x_1440x1080_-1n_mkv_geschnitten.mkv shows are due to computation errors, which usually appear if:
a. the system is overclocked
or
b. the system has heat issues
or
c. some other application is messing around with the memory (virus scanner or similar)
or
d. a hardware defect which causes issues during encoding and/or decoding

And it's basically impossible, that it is a system-problem.

No, it's just unlikely that it's a heat problem. Overclocking, etc. can still cause issues.
=> if your system RAM, fsb or gpu is overclocked, clock it down.

Aside from the Vapoursynth+sRestore issues with properly eliminating blends, this is not a general issue, but an issue that is due to something on your end.
(srestores decimation in avisynth works properly)

Cu Selur

[Platos]

I did a test with bwdif (avisynth) "double rate alternates each frame" + sRestore (avisynth). Then encoded, used this video in Waifu2x-extension-gui and i have the same artefacts. Dont know, if it's help, but yeah.

And i did a second test: With realEsrgan: Same problem...

[Platos]

The problem with sRestore and Vapoursynth atm. is that in the last quarter of the last year Vapoursynth updated to a new API.
This API broke some stuff that causes issues with sRestore, through which it will not properly decide which frames to reduce.
Only workaround I found so far is to call sRestore 2 times.

Ok, i have no hurry about vapoursynth. I just use Avisynth.

The artifacts Naruto DVD Folge 10_kurz2_avi sRestore + deinterlaced_waifu2x_1440x1080_-1n_mkv_geschnitten.mkv shows are due to computation errors, which usually appear if:
a. the system is overclocked
or
b. the system has heat issues
or
c. some other application is messing around with the memory (virus scanner or similar)
or
d. a hardware defect which causes issues during encoding and/or decoding

And it's basically impossible, that it is a system-problem.

No, it's just unlikely that it's a heat problem. Overclocking, etc. can still cause issues.
=> if your system RAM, fsb or gpu is overclocked, clock it down.

Aside from the Vapoursynth+sRestore issues with properly eliminating blends, this is not a general issue, but an issue that is due to something on your end.
(srestores decimation in avisynth works properly)

Cu Selur

Ok, i will try that with overclock. I dont have overclocked, i have undervoltet.

But question: Can you (or someone other) test on hybrid my source video with avisynth QTGMC+Bob (one time, normal in Hybrid) and sRestore (default Setting). Then encode and then the new video upscaling with RealCUGAN ?

If its not happen on your end with hybrid, i would have more input. It's only a 15s video, would be very nice. Should take like 30sec.

[Selur]

Attached:
1. deinterlaced, cropped and sRestore applied using Avisynth and encoded losslessly
2. reencode of the lossless source resized using RealESRGAN
2. reencode of the lossless source resized using RealCUGAN
going to bed now.

Cu Selur

[poisondeathray]

Ok, i will try that with overclock. I dont have overclocked, i have undervoltet.

Don't overclock. Don't undervolt. Set it default

The problem is definitely on your end. Something wrong with your setup, system, drivers ...

[Platos]

Attached:
1. deinterlaced, cropped and sRestore applied using Avisynth and encoded losslessly
2. reencode of the lossless source resized using RealESRGAN
2. reencode of the lossless source resized using RealCUGAN
going to bed now.

Cu Selur

Thank you very much. Also thanks for uploading the deinterlaced & sRestore file. Most usefull, because i could use it now to upscale it on my end with RealCUGAN. Result: No it has no Artefact. So i will try use srestore and deinterlace on my laptop gpu. There is everything on Default. Is even better, than de-undervolt my system. Because could also be driver problem. So i can test that.

I will update, if i have news.

Thanks all for your help.

[Platos]

Update: I did restart my computer and now it works. Strange... Did not even stop undervolting or sth like that.

Sometimes i just forget the most simple solution while thinking about complex problems.

Update 2: Today it happened again. I did nothing (went to sleep and now it did not work again).

Then i restartet Hybrid twice and then it worked again. Dont know what the problem is. It did not help to stop undervolting. Only the restart did help (after second restart).

Maybe i will find the problem someday.