Super-resolution techniques applied to Hi-Def

[koopak]

Hi everyone, boy have I got a good idea *awaits everyone to shoot him down*

I often watch/record movies that I already own on free-to-air television because they're in "High Definition" (1080p @15Mbps vs 576p @ 7Mbps typical).

Now I would just be happy with the High Def source, except for the following:
- There are ad breaks (that I cut out, but that leave fade in/outs and small sections missing)
- There are logos in the corner / pasted on banners advertising other shows
- There is insufficient bandwidth to avoid macroblocking in medium-high motion scenes
- Despite good signal strength, there are always the occasional lost packets

This has led me in the past to "splice together" sections of my DVD into the Hi-Def stream, which is painstaking to say the least, not to mention still noticeable in terms of varying resolution, and leaves the annoying logo.

I have come up with a brilliant solution, but I need assistance!

Take "Indiana Jones and the Raiders of the Lost Ark" as an example. I own the DVD ('perfect' 576p), I downloaded it in Hi-Def (logo-free 720p with small sections missing from ad cuts), and I have recorded it off TV (logo 1080p with small missing sections from ad cuts, but all different cuts than the 720p version).

Now if we just forget about the audio for a second (another article, another time, but it's also possible to apply the same process), it should be possible to have an algorithm compile a 99.99% perfect master from these 3 sources, here is my logic:

Algorithm/Program loads each frame of the 1080p and 720p sources 2 at a time (1 each) and compares them. Where the pixels are the same, they go through unchanged to new master copy, where they are different, pixels of the 1080p source go through to the master copy UNLESS the 720p versions pixels are of radically different value to the 1080p but are quite similar to the value of the DVD (576p) source, in which case the 720p's pixels are used.

Bingo, logos gone, advertisements gone, blockiness is reduced, ad cuts filled in!

And if you had TWO 1080p recordings, or FOUR recordings total, then you're laughing!

If the maths behind this seems suspect (and I am simplifying it), then check out a program called "PhotoAcute Studio", it does exactly this with stills, which is painstakingly slow for movies (designed for photographs), but which works surprisingly well!

PLEASE tell me this idea has already been implemented, laugh at me and tell me where the AVIS code is at!

Thanks and Cheers!

[filmboss80]

I'm laughing for different reasons. Think about the CPU demand and the rendering time required, even if such software existed. Wouldn't it be easier just to own the Blue-ray of the movie? Sure it costs a few bucks, but think of the time and sweat saved. Time is money, after all.

[guns1inger]

I'm laughing for different reasons. Think about the CPU demand and the rendering time required, even if such software existed. Wouldn't it be easier just to own the Blue-ray of the movie? Sure it costs a few bucks, but think of the time and sweat saved. Time is money, after all.

Apparently you can just download the 720p version from a legitimate vendor, so why not download the 1080p version instead ?

The first issue is that Super Resolution is just a fancy version of upscaling, and upscaling a 576p version to 1080p will not give you the same image. This means building a lot of fuzzy logic to cater for the differences in order to determine similarities. The you have the issue of replacing portions of an image or whole frames from sources that are substandard. Viewing a Super Resolution upscale of an image against the source or in isolation, if the upscale was done well, can certainly impress. Hold up against a true 1080p source, and the differences become very apparent.

You also have the issue that a simple frame by frame comparison is no possible in most instances. Taking your example, you have a DVD version, which is complete, but of inferior resolution. You have a 720p version which has ad breaks removed, so after the first ad break the frame numbers no longer align with the DVD source. Then you have a 1080p version which has different ad breaks, and therefore a different frame numbering. So now your software isn't just comparing frames to frames to clean up logos or replace fades, but must also be able to determine where a break occurs, and when it ends and re-align the frames to get back to frame - frame comparisons. And this all assumes that the three sources have the same framerate, and come from the same base master.

I am sure, at some point in the future, something like this will be possible. But at present, the sheer amount of power required to make it happen would make the days of leaving CCE running overnight to copy a DVD seem like a brief holiday.

[koopak]

Hmm, although the raw data rate would be large, I do not think the computation would be all that complex!

Surely there is something for AVIS that can take 3+ sources, and only pass on the value that holds the 2/3rds majority?

As far as frame rates go, that's easy, you just change them all back to 24fps, and as far as sync goes, I'm already having to sync the things manually at the moment anyway (to splice DVD footage into the Hi-Def footage).

I would love to buy the Blu-Ray copy, or download a 1080p version, the problem is that such things haven't been released yet.

Even if someone could provide me with a program that allows me to conveniently load 3 sources, play the main one until there is an error, and then allows me to pull frames from other sources to fix errors, it would be a great help.

I have seen in some video programs (such as Eyeon's Fusion) the ability to automatically sync 2 video clips, and it can deal with one of them being incomplete and not losing sync, so I know such things are possible.

Thanks for the thoughts tho guys!

PS: Super resolution (if done properly) is more than just a fancy upscaling technique. It's hard to explain without some diagrams, but by using the data from many frames, you can achieve a higher actual resolution (not just turned-up sharpness) than any one frame allows.

eg, lets say the full resolution shot has values like:

1 - 9 - 2 - 4 - 2 - 5 - 5 - 6 - 1 - 5 - 7

And we halve the res, so the frame now looks like this:

1 - x - 2 - x - 2 - x - 5 - x - 1 - x - 7

Which would be much blurrier, and guessing average values like normal upscaling would be wrong:

1 - 1.5 - 2 - 2 - 2 - 3.5 - 5 - 3 - 1 - 3.5 - 7 <----- Would look wrong, still blurry

But lets say, that the NEXT frame has this set of values:

x - 9 - x - 4 - x - 5 - x - 6 - x - 5 - x

Then if you apply super-resolution techniques (assuming no motion), you will now correctly have:

1 - 9 - 2 - 4 - 2 - 5 - 5 - 6 - 1 - 5 - 7 <----------Correct, identical to original

And since SO much of video has little movement from frame to frame, even BEFORE you use motion compensation techniques (to track objects like cars or faces across the screen, lining them up for comparison / detail adding), you are able to recover quite a lot of actual resolution that spatial-only upsizing techniques could never recover!

[guns1inger]

You can't just convert them to 24 fps. You have telecine issues, audio sync issues, etc. Depending on how you deal with some of these, you can get impacts on the image data.

Which ever way you cut it, Super Resolution results in manufactured data to fill out what never existed to begin with. I understand how it works, and I still stand by my rather glib description of it. You will be comparing manufactured image data to actual image data. The lower the resolution of the source, the more data has to be manufactured, no matter how that data is created.

If you have every used Fusion at HD resolution, you would understand that even on a quad core system, it is very slow going.

[filmboss80]

There is also the question of ownership of a movie copied from a free-to-air broadcast. Since the broadcasters offer it free, they have the right to place their logo bug in the picture. If you copy their signal, you should accept their logo bug and not tamper with it. It seems like what you are proposing is actually an extremely convoluted and roundabout form of warez. Again, I say buy your own copy.

[koopak]

Hmm, or maybe I'm a guy with a large LCD who is demanding of quality and would BUY a copy if he could (and already owns the VHS, DVD AND Blu-ray copies of many a movie), but who is willing to invest time and effort in obtaining the best quality available.

I AM using Fusion on HD data, on a mere dual core, on data from an EX1. I have several computers, and have been used to long render times since Quad Pentium Pro days on Lightwave 3D. The network render function helps since i have 10 Core/Core2s available.

I am in no way saying that super-resolutioned data is better than having the raw uncompressed full-res data, and that would be an insane argument, which I'm NOT making. What I'm saying is, if you HAVE to make guesses about missing data, you might as well use the best method available, and with it's temporal data gathering, super-resolutioning most often is the best? (My comparisons to Spline36 have been extremely favorable to SR, that's why I'm biased)

Anyway, rather than starting a flame war (and I unequivocally state here that I am against warez, and am interested in quality, not piracy or penny pinching), is there anyone who knows how to do "best of 3 (or more)" image analysis in an automatic way, either inter-image or intra-image? I'm not fussed about the technique, if super-resolution is such a 'defies-laws-of-physics-lets-pour-scorn' method, I'll gladly use a better one.... now what would that be....?

Last but not least, this has wider reaching applications, forgetting copyrighted media. The ability to take two bad copies of something to make a better copy would be possible and extremely useful. I can think of 2 home movies already that would appreciate this technique (originals lost, bad VHS copies only remain).

Any help appreciated!

PS: With my 3 sources, there is no problem with frame conversion:
25fps footage is just sped up 4% from 24fps, but is still exactly the same frame for frame as the original, you just slow it down. I'm not working with NTSC, so there's no telecining to consider. And even if there was, this isn't anime (yes I do plenty of VFR conversions, been banging my head against the wall converting Avatar / Dilbert for the past month), it's a pretty easy process to go 30 -> 24 these days for film!