Using Nuke to Upscale

[SameSelf]

So I know there are lots of ways to upscale: Lanczos, Spline36, and so on. But I recently learned about Nuke which has a resizer called TVIScale. The Foundry has a free non-commercial version that is only mildly crippled. So I thought, why not? I downloaded it, installed it, and tried it out on one of my vids. Took some time getting used to because I have never used a node based program before. All I can say is I am sold! This thing is super powerful. For the first time in my editing experience I don't feel like everything is a black box. If I get some time I might post back with some stills of the resizing algo's I am using.

Anyway, just wanted to see if any else here has experience using Nuke to uprez.

[LigH.de]

If you had posted a link to the website of the producer, we all would have had a chance to participate in testing...

This node scales an image by a factor of two. It uses a Total Variational Inpainting (TVI) technique that minimizes noise while still preserving the edges.

[SameSelf]

Google

[LigH.de]

Okay, then Google for "Nuke" and tell me at which position you'll get a hit for a video framework ... telling people to use a search engine is not sufficient, even cynical, without giving hints how to narrow the results. Without knowing that "The Foundry" is the name of the producer, I would not have found it.

[SameSelf]

I left more than enough clues for the mildly intelligent to find it. But my post is more geared towards soliciting comments from current users.

[poisondeathray]

I use NukeX, but not really for upscaling. Maybe it's time to revisit it

Have you compared to nnedi3_rpow2 + mild sharpening ?

Are you using it alone (single node) ? Or how do you have the script setup ?

Looking at other user posts - you're supposed to use it with other nodes and operations for best results - but that's going to vary by source of course. eg. how much to denoise, how much to sharpen, what matrix to use for AA etc...

Also what looks "good" on a still image doesn't necessarily translate to video. The danger with any inpainting based technique is they are prone to temporal fluctuations between frames.

And sort of related to the discussion, but any super-resolution based upscaling technique can be prone to temporal artifacts

[SameSelf]

I use NukeX, but not really for upscaling. Maybe it's time to revisit it

Have you compared to nnedi3_rpow2 + mild sharpening ?

Are you using it alone (single node) ? Or how do you have the script setup ?

Looking at other user posts - you're supposed to use it with other nodes and operations for best results - but that's going to vary by source of course. eg. how much to denoise, how much to sharpen, what matrix to use for AA etc...

Also what looks "good" on a still image doesn't necessarily translate to video. The danger with any inpainting based technique is they are prone to temporal fluctuations between frames.

And sort of related to the discussion, but any super-resolution based upscaling technique can be prone to temporal artifacts

Hi pdr, that's some truly incredible insight. Thanks for the reply. I have done some mild comparisons but nothing real thorough to test the temporal fluctuations you mention. The rendering step is slow and seems to default to intermediate codecs I have never heard of. For me the gold standard is my TV because it is only then that I see stuff I never see on my PC.

I am aware of nnedi3_rpow2 + mild sharpening, having used it myself in the past. There are numerous threads out in the wild comparing the two. The best thing that seems to be said is you have to just try both and see which one works best for your footage. The NC version of Nuke cripples at FHD. So I am taking 480i video and uprezing to 960p. I should also mention that I push the 480i video through a QTGMC filter before bringing into Nuke because Nuke doesn't play nice with interlaced footage from what I read.

As for my nodes, I do a mild blue channel blur, convert to YCbCr and do a mild RG channel blur before scaling. Afterwards I put in an AA matrix and sharpen. It is a total of nine nodes with all the YCbCr <--> RGB conversions. I am a little leary of posting my script because I am far from an expert here.

This is very experimental at this stage. So feel free to comment. I wish I had some stills to post, but it is just too early at this stage. But the other great thing is Nuke seems to offer some POWERFUL denoising nodes on the line of NeatVideo but out-of-the-box. This could be a HUGE boon to my HD footage. So much to learn, just not enough time!

[LigH.de]

Another upscaling tool for single images I know is Benvista PhotoZoom Pro, but I believe it uses only "smooth" algorithms based on splines; and I doubt that e.g. "S-Spline Max" is available as video filter, either...

An implementation of TVI for AviSynth may be interesting; but it will probably depend a lot on pre-filtering if such temporal fluctuations can be suppressed.

[poisondeathray]

Yes, Nuke is very powerful - it's like After Effects on steroids .

I was going to post some comparisons, but it's difficult to make "apples to apples" comparisons of upscaling algorithm *only* , because the process usually involves other operations like channel manipulation, denoising, sharpen, deconvolution, antialiasing etc.., and you are going to use different filters, settings, and techniques for different scenarios. So you're not evaluating the upscaling filter alone, but rather an entire chain of filters or entire workflow. It's like that for almost all "upscaling" filters, commercial or free - they have dials or sliders for a bunch of settings, not 1 size fits all - whether they are "photo" based or "video" based ones

I've tested a bunch of them, including PhotoZoom , Topaz Enhance, Video Enhancer, Adobe's detail perserving upscale, Red Giant Instant 4k, BCC Uprez... many more... They each have strengths/weaknesses. The standalone versions don't have the versatility of nuke or after effects or even avisynth for accessory operations

Some general observations of this filter alone - it doesn't have AA built in - that already causes buzzing edges independent of the inpainting (but you would never use it alone) . It does a good job of peserving more detail (perhaps false details in some instances). Might be too sharp at default values, and there seems to be border issues (frame border edges, not content edges). It's definitely not a 1 or 2 click wonder like some of the other commonly used software for upscaling, you definitely have to play with other nodes to get better results

[SameSelf]

Yes, Nuke is very powerful - it's like After Effects on steroids.

That was my first impression too. This is a tool that in the hands of someone who is competent is incredibly powerful. At the very least, it unobfuscates the black box nature of AE and Avisynth in a much more intuitive manner (the only tools I have used). IOW, I am a big fan of the node workflow now as it goes a long way of helping me to understand how each of these operations affects the final result. Now, I just need to find a spare $2,500, cough cough.

[poisondeathray]

avisynth might be a "black box" to many people, but I would argue AE is by far the easier one to use for most people, even easier than Nuke . AE is the most intuitive because it's layer based , like photoshop. All the same image manipulation concepts transfer over. Nuke is favored by power users, but for general use compositing and manipulations, AE is definitely faster and easier to use

You can also have a look at natron and blender which are free, open source nodal based software. Natron essentially a nuke clone (the UI is almost identical), blender is more oriented to 3D but still has many of the functionality of a compositor and some NLE capabilities in one

https://natron.inria.fr/
https://www.blender.org/

[SameSelf]

avisynth might be a "black box" to many people, but I would argue AE is by far the easier one to use for most people, even easier than Nuke . AE is the most intuitive because it's layer based , like photoshop. All the same image manipulation concepts transfer over. Nuke is favored by power users, but for general use compositing and manipulations, AE is definitely faster and easier to use

You can also have a look at natron and blender which are free, open source nodal based software. Natron essentially a nuke clone (the UI is almost identical), blender is more oriented to 3D but still has many of the functionality of a compositor and some NLE capabilities in one

https://natron.inria.fr/
https://www.blender.org/

Thanks for the tips which I should have said sooner.