Higher resolution

[Oleander]

Dear all
I have a video with low resolution and i want ro capture a face of a lady in the video with higher resolution. Can anyone help me?

[jagabo]

No.

[johnmeyer]

To expand on jagabo's correct answer, you cannot extract detail that isn't there in the original. All those scenes in movies where someone asks, "Can you enhance that?" are 100% fake.

[yboris]

It is possible, but at the moment requires you get into some programming.

Machine learning is making extraction of higher resolution possible by estimating underlying source by comparing adjacent frames of video.

This is the most successful thing I've come across recently: TecoGAN https://github.com/thunil/TecoGAN

[poisondeathray]

@yboris - have you tried TecoGAN on actual video yet ? Do you have any examples you can post ?

It looks interesting, it's one of the few that actually look at other frames

Most GAN's are trained on still images, so the results aren't that great on "regular" live action content (can work ok for some types of cartoons, still photos for some types of textures) , and noise is usually a big limiting factor in real video.

In general , I haven't seen very good results with people / faces . Typically you can good results with things like wall textures, ground textures, foliage

[Cornucopia]

Here's the ultimate truth regarding restoration:
you CANNOT create detail (aka valuable information) out of nothing. You also can NOT create MORE detail out of minimal detail.

Without *much* WORK. (Much like the laws of thermodynamics).

This work is usually artistic work, but could be a combination of that and engineering work.
If it is solely engineering work, and automatic at that, it just amounts to a bunch of guesses.
IMO, not "valuable" information.

Silk purse, sow's ear.

Scott

[johns0]

I bet in the future when we have much better a.i. in looking at a picture/video and it knows what it should look like in a better quality and fixes it but that's years from now.

[poisondeathray]

For photos, the "guesses" or "detail creation" are pretty good for cetain types of textures. Some have weird artifacts, but it's still early, and you can interpolate between 2 models (eg. an aggressive model , and a conservative) to get something in the middle

If anyone has looked at Topaz AI Gigapixel, it does pretty good on the types of stuff metioned above in photos. Not as good on faces, or video (video has noise, artifacts, motion blur, chroma subsampling).

We've all see AI generated faces - some are pretty realistic.
https://www.wired.com/story/artificial-intelligence-fake-fakes/

You need specific models to get good results. If you use a generic model, the results won't be as good

So we need a trained model dataset specific for faces and people in specific "video" conditions that takes into account motion blur, lossy compression artifacts, multiple frames, temporal consistency. That TecoGAN looks like it's a step in the right direction for video

EDVR looks good too. Same author as ESRGAN
https://xinntao.github.io/projects/EDVR
https://xinntao.github.io/open-videorestoration/

Many of the GAN research projects on Github are python based, and some of the projects and models are usable right now in vapoursynth

[Cornucopia]

Those only have the possibility of appearing "realistic" if you don't have a familiar reference to compare them to.
(Particularly in an A/B shootout comparison).

If you tried to create a hirez picture of YOUR Aunt Jean from a lowrez thumbnail, you'd know immediately that these attempts are (for the forseeable future) bogus and ineffectual.
Don't forget the Uncanny Valley in the Realism Curve.

Scott

[yboris]

EDVR looks good too. Same author as ESRGAN
https://xinntao.github.io/projects/EDVR
https://xinntao.github.io/open-videorestoration/

Thank you poisondeathray! This is AMAZING !!!

[poisondeathray]

Those only have the possibility of appearing "realistic" if you don't have a familiar reference to compare them to.
(Particularly in an A/B shootout comparison).

As mentioned earlier, for things like foliage, brick walls, landscape shots etc.... Those work quite well - because current models have been trained on those situations. Repetitive textures, or conditions similar to PS's content aware fill or inpainting work well

Certainly they can much better than a typical "upscaler", but also some weird artifacts in some cases (not perfect, but when they do produce good results, or you mix models, it can be 10x better than an upscaler)

The "reference" is the ground truth ; a higher resolution version which it has been trained against. That's how these GANs work

When you apply it to other photos, other situations, the "guess" is based on the training .

But most the training is based on degraded, lower res versions, but the low res versions are still quite "sharp". They don't have the motion blur, chroma noise, compression artifacts etc... that video typically has. That's why they don't do so well on video. Noise is usually the limiting factor


I'm quite a skeptical person, but this is the real deal (at least right now for some types of photos, some types of cartoons)

If you tried to create a hirez picture of YOUR Aunt Jean from a lowrez thumbnail, you'd know immediately that these attempts are (for the forseeable future) bogus and ineffectual.

As mentioned earlier, specifically people/faces are more difficult . Special training has to be done for that situation (and in video situations). But these GANs are getting better at creating faces, and those types of characteristics (decisions it accepts or rejects) will be similar to the ones they make for upscaling faces