so where's all the Topaz Video Enhance AI discussion?

[lordsmurf]

this new fangled AI gubbins.
The videohelp mullahs have decide AI is not welcome here, so why bother?

AI = artificial intelligence. There is no AI here. It's marketing.

Even nnedi3 claims to use "neural network architecture", but is really nothing more than predictive (aka opposite of reactive) algorithms.

"Neural networks" are a basis for AI, and the name "neural" comes from mimicking human (or any animal, really) brains. There is parallel processing, but there must also be an element of development. That would insist on the ability to save data for later analysis. And with Topaz, or nnedi3, or others, that isn't happening. There's no learning file being built.

So the "AI" here is dumber than an amoeba.

In reality, what happens here is that a pre-made "bank" of set data has to act as the basis of prediction, and it does not learn anything. And that's nothing new, having existed for decades. I think back to OCR scanning in the 90s, or speech-to-text in the 00s. Google Voice is probably a good example of actual (limited) AI, because the algorithm was allowed to grow, and has become far more accurate over time.

https://www.ibm.com/cloud/blog/ai-vs-machine-learning-vs-deep-learning-vs-neural-networks
https://www.bmc.com/blogs/neural-network-introduction/
https://www.verypossible.com/insights/machine-learning-vs.-neural-networks

When Topaz can make a T-1000, then maybe it'll be AI.

[Cornucopia]

What bugs me about all these "comparisons" is that there is NEVER a Ground Truth reference to any of them.**

If they were doing it right, they would have:

1. Multiple types of source material (real-world, CGI and cell animation, high-con & low con, test patterns, very high quality & low quality), all which are lossless (THESE are the Ground Truth references).
2. Downscaled lossless versions of those.
3. Detailed listings of the apps, workflow & settings used to create the "upscaled" versions. This is imperative to establish experimental repeatability, which is fundamental to any peer-reviewed scientific model. Especially one that is considered a "breakthrough".
4. Upscaled (lossless) versions of the downscaled sources (#2). And you would compare the #4s against not only #2 but also against #1.

This way, you know whether an upscale is getting closer to "reality" or not. Otherwise, it could be generating interpolations that might at first glance appear to be reasonably clean & smooth and possibly "natural", but in comparison are wildly synthetic (and possibly more difficult to compress?).

Notice that NONE of those used lossy compression. That's its own can of worms.

We always seem to be given one or 2 curated instances of #2 (and usually compressed) along with their #4s, but NEVER any #1s and rarely any #3s. That's disingenuous as far as the scientific method is concerned.


Scott

** I see this in Upscaling tests, Colorization tests, 2D->3D tests, mono->stereo->surround tests, NoiseReduction tests, compression tests...

[lordsmurf]

What bugs me about all these "comparisons" is that there is NEVER a Ground Truth reference to any of them.**

You'll also find that many comparison tests are "simulated" -- aka, total BS.

[johns0]

Also ai means that the programs learns and does it?Never been shown to improve anything,just the same old output settings.

[Selur]

Neural networks only learn if their dataset/model is updated, otherwise they make their decisions based on the dataset/model.

[brandon87]

Here's an excellent example of VEAI in action. It's copyrighted, but free to share (nrk.no gig). Note it's on 1337x, so if you live in India, don't click:
Aurora - 2017-11-02 Nidarosdomen, Trondheim, Norway v2.0 (1080p50fps)

File size: 10600MB
Overall bit rate: 25900Kbps
Video: AVC1 High@L4.2 (H264) 1920x1080p 50.000fps 25700Kbps
Audio: AAC 160Kbps 48.0KHz stereo
Total time: 00:58:38

Processing details

[DVWannaB]

Thank you jagabo for the Avisynth upscaling script. Tested it in on good quality VHS tape upscaled to 720p (that's all I need for VHS conversion). Loving the results. My lone contribution to this thread is that if you are committed to your task and willing to learn you can achieve a high level of success. In this case Avisynth can be a pain in the rear to learn, scripting is so 1990s (lets change this already), but its free and works. All I do is mimic the work other guys have done (jagabo, in this instance) and tweak as necessary. VEAI is not bad, but I am not convinced it provides me with a greater quality difference I get from Avisynth/Virtualdub combo. To each his own. Live and let live.

PS. This product is NOT using AI. That's a gross misrepresentation of what AI is.....LOL

[lordsmurf]

Here's an excellent example of VEAI in action.

Not really. CAS() seems to be the reason for sharpening.
I'd wager nnedi3(rpow2)+CAS would yield even better results that Topaz+CAS.

It wasn't your intention, but thanks for the post, I need to investigate CAS more.

[Selur]

Yes, CAS is nice when upscaling, just make sure too not always blindly use it with strength 1.

[lordsmurf]

Yes, CAS is nice when upscaling, just make sure too not always blindly use it with strength 1.

Noted.

[Selur]

Also it will enhance not just details but also artifacts so best only use it on clean sources.
But you'll see that if you play around with it a bit.

[dolpsdw]

Hi, i am attempting to restore "Sesame Street", i am new on the field.

I found this threat because i have been experimenting with topaz a bit.

Is this kind of "enhancement" possible with more manual approach ?

I clearly see "invented" details on the right image.

The thing is clearly not perfect and could be considered as "profanation" of the original signal, but still i think my kids could enjoy it more with those fictional details.

My main question is: could i achieve similar results with more manual tools?

Pdta: Original video -> https://www.rtve.es/alacarta/videos/barrio-sesamo/barrio-sesamo-prueba-vestuario-chelo...llejo/4679684/
Upscaling + enchance only with Topaz, no QTGMC.

[johnmeyer]

I see absolutely no magic and no new details in that before/after pic.

Here is a before/after on some old film. I've posted this many times before. I did this with two sharpening tools, and nothing else. The sharpening code is part of my film restoration script, posted over at doom9.org. It is nothing unusual or special, although I spent man-days of effort tweaking the settings to get it just right (i.e., improvement without too many new artifacts).



If you look at the vertical support posts on the porch railing in the background, you might be tempted to believe that there are "new details" that have been "invented," but that is not true. It is just sharpening.

If AI "Super Resolution" is ever going to be convincing, it will have to fill in the lines on a resolution chart which are totally missing.

[mammo1789]

It's overpriced garbage. The "AI" marketing suckers newbies. You've been had.

It's just an upsizer, with sharpening, and Avisynth does better.

Topaz is/was a low-end Photoshop plugin company. Their stuff works, but always worse (sometimes MUCH worse) than slightly more manual methods.

A lot of those DS9 Topaz examples really suck, lots of artifacts.

I must agree that it is total garbage especially for home made vhs sources. It makes picture sharper but introduces some nasty artifacts that are not there in the original. It was fairly most successful with some low res phone videos other than that not worth the money

[lordsmurf]

Also it will enhance not just details but also artifacts so best only use it on clean sources.
But you'll see that if you play around with it a bit.

Yep, noticed that. I played with it this morning.
I didn't realize that you'd already included it in Hybrid!
Total accident to test today. Was doing something else, then saw it in the Vapoursynth area of Hybrid, got sidetracked. I wasn't intending to test CAS, but couldn't help myself.

The sharpening code is part of my film restoration script,.

You and I butt heads sometimes, but that script ... a work of art.

If you look at the vertical support posts on the porch railing in the background, you might be tempted to believe that there are "new details" that have been "invented," but that is not true. It is just sharpening.

I was doing this almost 20 years ago. I still remember how slow processing a video was, just for a few filters.

If AI "Super Resolution" is ever going to be convincing, it will have to fill in the lines on a resolution chart which are totally missing

Agreed!

[johnmeyer]

You and I butt heads sometimes, but that script ... a work of art.

Coming from you (a person who's work I very much respect), that is a real compliment. Thanks!

I was doing this almost 20 years ago. I still remember how slow processing a video was, just for a few filters.

Almost all the performance gains in that timespan have come from computer parallelism; the clock speeds have not increased. That parallelism didn't help much for AVISynth processing until the developers figured out ways to multi-thread some (not all) of the filters. At that point we started getting 4x-6x speed gains.

I need to continue to update (which I have not done) to newer versions of AVISynth which I think do an even better job of taking advantage of multiple cores. However, I am doing less media restoration these days, so I haven't had the compelling need to invest the 1-2 days it will take to update everything.

[goehler]

Wow. This was an interesting thread.

A lot of headbutting by people who were obviously both passionate about the topic and frustrated with each other. :P

I stumbled into this, trying to find some guides for Topaz, that I'm currently using a 30 day demo of.
I will say, I get some quality improvement, but not as much as I would prefer. And I see the artefacts that some of the Topaz methods clearly introduce.
I'm very impressed with John's result above. That enhancement is really superb - and when I have some time to dive into Avisynth.

@johnmeyer - Do you have a link to your mentioned script? I'd like to visit that some time when I know a bit more.

[johnmeyer]

@johnmeyer - Do you have a link to your mentioned script? I'd like to visit that some time when I know a bit more.

My script is a fairly major re-write of the original work done by "VideoFred." His script was designed for pristine film, just back from the lab. My adaptation is tuned for old amateur film that has major defects. I removed a lot of the sharpening in Fred's script because it tends to emphasize the flaws in the old film, but I wanted to retain the "magic" that VideoFred developed with his sharpening. I think I mostly succeeded.

Here is a link to a somewhat recent version of the script. It is quite long, but you can extract the portions you find useful:

https://forum.doom9.org/showpost.php?p=1861734&postcount=1045

If you are interested, here is a before/after video I've posted many times which shows what this script can do:

Film Restoration- Before After

[brandon87]

The VEAI 1.8.0 beta was just released. I skipped 1.7.1 due to major issues with it (blocking artifacts and lack of good models), but with 1.8.0 beta, they have made much progress on the Artemis-LQv9 and Artemis-MQv9 models. I have a 480i film DVD that has so much noise, that I reverted to Artemis-MQ in 1.2.0 to deal with it (it was removed after 1.2.0). With FluxSmoothST on the pre and post Avisynth scripts, I was able to get a decent (not perfect) 1280x720p. And with 1.8.0, I hope I can make it perfect. I have 2 VEAI boxes now each with a GTX1080Ti, one with a 4.9Ghz 8 core CPU + a fast Xeon 80TB file server. All are working on creating the new video using 1.8.0. I'm doing 3 tests with the film: Artemis-LQv9 at 400%, Artemis-LQv9 at 200%, and Artemis-MQv9 at 200%. Tomorrow evening all should be done and ready for review.

Somewhat related: I found that combining CAS with vsMSharpen (in that order) is the most powerful sharpener I've used. If CAS sharpness is too high, it'll bring out the noise. If vsMSharpen strength is too high, it'll make light or white halos on the edges. Used with the right settings, it makes a big difference in the detail seen without oversharpening.
For example, in my post VEAI script I might add after downsizing to the final resolution:
CAS(sharpness=0.7)
vsMSharpen(threshold=6.0, strength=30.0, mask=false, luma=true, chroma=true)

I encode the mp4 and watch. If the settings are right it'll look great. If not, I'll rinse, tweak, repeat until it does look great.

Guidelines I use:
I first try to remove noise with QTGMC. If that fails, I'll do no noise processing with QTGMC and try other means- FluxSmoothST has worked best for me.
If no noise high quality 720p or 1080p, I use Gaia-CG from 1.6.1.
If noisy, I use Artemis-MQ from 1.2.0. This works best with less than 720p sizes.

Post 1.8.0 beta guidelines:
Anything with noise I'm trying on Artemis-LQv9 and Artemis-MQv9. You can see in the LQ preview if it's too smooth, that MQ might leave more detail.
I'm trying Artemis-HQv9 with a high quality 720p25fps vid I just completed an excellent 1080p50fps upscale on using Gaia-CG 1.6.1 (not the Nidarosdomen vid, this is a different vid). I have like 10 film DVDs to redo now that I've used 1.8.0...

1.8.0.0b is free for 30 days. Go to the facebook group, on the Announcements section, click See All to get a link for 1.8.0.0b:
https://www.facebook.com/groups/554187698731504

[lordsmurf]

My script is a fairly major re-write of the original work done by "VideoFred." His script was designed for pristine film, just back from the lab. My adaptation is tuned for old amateur film that has major defects. I removed a lot of the sharpening in Fred's script because it tends to emphasize the flaws in the old film, but I wanted to retain the "magic" that VideoFred developed with his sharpening. I think I mostly succeeded.

In turn, I've adapted it to lousy film>VHS conversions from the 80s/90s. But I have a hard time working with your scripting style, too many variables, backwards to how I learned Avisynth (over a decade ago). So mine was mostly to remove and scale back some aspects, which overemphasized issues similar to what you did to Fred's script. And then I have a pre-processing script, to attempt to yank back the film out of the VHS. It's nothing I've shared yet, it's still messy, and so far only used a few times.

Video like this would have entirely butchered by Topaz software.

While my end product isn't as nice as johnmeyer's output, remember that it was massively damaged by the VHS middle step. So if I'm able to even come close to what he's done on his, I've succeeded nicely. As much as my additions were needed, for my sources, the magic is still the core of script (with all deference to Fred, when his input still remains).

Credit where credit is due.

[Corvius]

Nobody here cares about it cause its just another ai so called up sizer.

Machine learned based algorithms are now actually capable of producing impressive results, it is the way of the future. Anyone who denies this shows that they are just another old timer clinging to their past tried and true techniques and are probably a lost cause, having completely shut out emerging technology from their perspective.

[Corvius]

Original Frame w/ Lanczos: https://postimg.cc/mPmxYW9L

Resized in Topaz VEAI 1.2.0 using Artemis-MQ model: https://postimg.cc/XZqRx1Kw

[Cornucopia]

Notice - continued use of #2 vs #4, both compressed, without reference to #1 or #3 as I mentioned above.

BTW, here is a contrast-expanded version of the differences.
I see lots of edge enhancement. Too much - haloes.
Color saturation changes.
Also some mottling in areas there shouldn't be any. Almost looks as if the Lanczos version were de-tweaked.

But we'll never truly know without #1 or #3. I'm not saying it's not possible to improve on existing methods, I'm saying how can anyone accurately and scientifically compare when given curated show ponies, no reference and no secret sauce recipe?


[Attachment 56295 - Click to enlarge]


Scott

[Corvius]

Source video was NTSC DVD with the standard telecine you see in such DVD's. It was processed using Selur Hybrid: frame decimated using TIVTC in AviSynth, re-encoded as .mp4 using CRF of 1, then upscaled with VEAI, version 1.2.0, using the Artemis-MQ model in that version. That's it.

Don't understand what looks to me like just outright hostility. Why not use all the tools available to you?

By the way, the one thing I agree with is that "AI" is a completely wrong term to use here. There is no "AI" anywhere, these are machine learning based techniques... algorithms that have been "trained" with source data to respond in a more specific manner given a similar source.

One thing that is also true, is that machine learning resizing is going to give the best results on material the most closely matches what material a given model was "trained" with.

The Artemis-MQ model in VEAI 1.2.0 looks like it was trained using filmed sources that were scanned in HD then resized for standard definition. When using a source DVD that was mastered very well (scanned in HD from a film source), you will get really good results because commercial DVD's are typically encoded using a very high bit rate.

If you process video with VEAI that doesn't really match up to the training data, then it won't find anything to process, and the result will look mostly like a regular lanczos resize with some sharpening added. In some cases it may "find" parts of the video that match training data and create some detail just in those areas.

The program has only been around for a year so it's still in its early days. It can only get better.

I have nothing to do with the company. I don't work for them and am not part of any kind of marketing firm, or have any kind of commercial affiliation with Topaz whatsoever. I like what the software has been able to achieve so far. I use it in conjunction with AviSynth, Hybrid, QTGMC, TIVTC, Derainbow, DFM Derainbow, Auto Levels, LSFasterMod, to name a few. I'm not going to NOT use an available tool or resource due to some kind of irrational hostility or principle. I try things, if I like what I see, I use it. If it costs money, I'll pay for it if I like it enough.

[Cornucopia]

"Outright hostility" - where? WHAT did I say that impinged on you personally?

Did you supply the sources (e.g. 10 sec DVD clip using the usual VH recommended lossless methods)? No
Did you supply all the steps? Maybe partly, but there's no way we can re-create without the source. And "processed using Hybrid" means a lot more than those few items. Actual screenshots of what settings were used at each step is the way to do this honestly in a scientific, evidence-based manner.

I'm not saying you're wrong, or bad, or whatever.
But you're making a claim without providing the details that are needed to be backing it up for proper peer review. Do you have something against giving us that actual data?

Do I harp on this with others making similar claims? Most of the time, when necessary, and especially when the claim is more inflated.


Scott

[johnmeyer]

"Outright hostility" - where? WHAT did I say that impinged on you personally?

Yeah, I didn't see anything either. I even went back and skimmed earlier posts from other people. There was certainly hostility towards the OP, and I posted about that back on the first page, but everyone was helpful after that, so I don't know where he's coming from. You certainly were not hostile in any way whatsoever.

[Corvius]

There was certainly hostility towards the OP

That's what I was referring to, as well as to VEAI in general.

[manono]

I didn't see anything either.

Yeah, me neither. I backed up to read the 2 previous posts of Corvius and the first thing I thought was, "Two pictures are supposed to be proof?" And the second was, "Where's the source DVD video?" Cornucopia's points are valid and in no way express any hostility at all. Had I cared enough about this VEAI to post a detailed comment, I might have shown some hostility.

Edit: Corvius posted while I was reading, looking and writing.

[lordsmurf]

The program has only been around for a year so it's still in its early days. It can only get better.

No.

You lack context. I bet you never heard of Topaz Moments, have you? It's a Topaz software from 10+ years ago, for both still and video. Lots of typical Topaz claims of "bestest ever" upsize/sharpen, blah blah blah. But it never happened. We got a beta-quality release, the end. No real improvements ever. You'll have a hard time finding any info on it, virtually scrubbed from the interwebs (more proof that "anything online stays forever" is bunk).

There's been a long list of broken-promiseware, vaporware, and disappearedware, in the video space, going back decades.

All I see here from Topaz is marketing, suspect "samples".
And adoption by people who are impressed by ringing/halos and other artifacts, using sources like Youtube and retail DVDs.

Don't understand what looks to me like just outright hostility. Why not use all the tools available to you?

I'm hostile to outright bullshit. But mostly because of this...

By the way, the one thing I agree with is that "AI" is a completely wrong term to use here. There is no "AI" anywhere, these are machine learning based techniques... algorithms that have been "trained" with source data to respond in a more specific manner given a similar source.

And you agree. So ... yeah.

One thing that is also true, is that machine learning resizing is going to give the best results on material the most closely matches what material a given model was "trained" with.
The Artemis-MQ model in VEAI 1.2.0 looks like it was trained using filmed sources that were scanned in HD then resized for standard definition. When using a source DVD that was mastered very well (scanned in HD from a film source), you will get really good results because commercial DVD's are typically encoded using a very high bit rate.

I'm not sure I agree with this.

If you process video with VEAI that doesn't really match up to the training data, then it won't find anything to process, and the result will look mostly like a regular lanczos resize with some sharpening added.

I've really not seen anything that doesn't.

In some cases it may "find" parts of the video that match training data and create some detail just in those areas.

This has lead to artifacts.

I'm not going to NOT use an available tool or resource due to some kind of irrational hostility or principle. I try things, if I like what I see, I use it. If it costs money, I'll pay for it if I like it enough.

Same here. But I'm seeing zero results from Topaz that would make me want to use it, free or paid. Perhaps this will change, perhaps not. Likely not.

[litmus]

This thread amuses me. I had a similar negative response from the forum elders when I posted about an upscale project I was doing for Star Trek Deep Space Nine using TVEAI back in April. https://forum.videohelp.com/threads/396652-Deinterlacing-Deep-Space-Nine-NTSC

Funny how people keep emerging from the woodwork as proponents of this software which there's absolutely nothing to it eh?