[Avisynth/VirtualDub] Vibrating frames and white speck removal.

[LaKap]

QTGMC won't get "rid" of the blending, it will probably introduce more blending (The "T" in QTGMC stands for temporal)

Between yadifmod+nnedi3 and QTGMC(), qtgmc worked better. I don't have much experiecne with MVTools so not sure what to use exactly. Will be reading the docs and testing it out tonight. What would you recommend?

Well it's distracting to me, but your average viewer might not notice it that much. Avergae viewer probably don't even notice dirt/scratches that much. So I guess it depends who your audience is, and their expectations

My aim is to restore it to a better state that is achievable with the given input.

I can outline how I would approach it if you like

That would be quite helpful actually .

[poisondeathray]

QTGMC won't get "rid" of the blending, it will probably introduce more blending (The "T" in QTGMC stands for temporal)

Between yadifmod+nnedi3 and QTGMC(), qtgmc worked better. I don't have much experiecne with MVTools so not sure what to use exactly. Will be reading the docs and testing it out tonight. What would you recommend?

Not sure for the blends, I'll take a closer look later. But QTGMC can be problematic for blends because of it's temporal nature - it can take the adjacent blends and ruin "good" frames.

My aim is to restore it to a better state that is achievable with the given input.

That's a very subjective statement. There is a continuum of "better", probably roughly relative to amount of effort put in . For some people, "good enough" occurs very early. Others obsess over every detail

I can outline how I would approach it if you like

That would be quite helpful actually .

There is a lot to go over, but I'll throw out some general concepts:

The reason why standard stabilizers won't work for the "vibrating" is they typically adjust x,y and scale parameters globally (ie. the entire frame is counter shifted x pixels, y pixels and or +/- scaled) . But your video has parts of frames that move in different directions, while others are static. It's "morphing" sections of the frame non linearly. View it frame by frame to get a sense of what is going on . Therefore, global shifting of the frame won't fix it

So the repair involves compositing techniques. For example, the 1st video "vibrating" would require you to roto out the foreground actor (various techniques & tricks are used to speed this up and achieve separation, because manual roto work is the bane of FX work - again there are hundreds of tutorials and approaches on this concept alone) , and composite onto a stabilized background. That stabilized background can be generated from various techniques, even avisynth (e.g. temporal stabilizers) . The actor needs to be filtered differently to preserve the details (you can overfilter the background, because of depth of field - it's already "blurred", but more attention has to be played to foreground elements). Notice that different areas of the frame move differently (ie. morphing) . Motion tracking is used for specific elements "patch" over sections locally , and also globally to reapply the natural camera motion . The 2nd video is more complex, and involves more camera movement, but also falls under the category of "patch" repair . I've posted a bunch of info on motion tracked patch repairs before. You're basically replacing elements in the video with different "patch" layers , all linked together with motion data .

"Interpolation" between two points was mentioned earlier - e.g. with mvtools2 . Basically you're replacing over "bad" frames from two known end point "good" frames. Those "inbetween" frames are replaced or "interpolated" over. Some repairs can be sped up considerably with this method, or at least partially sped up and repaired (with touch up in AE or photoshop). Many avisynth filters and approaches are invaluable for part of the repairs - and you combine them with masks and compositing. You also limit the damage caused by them by being more selective. It's all about applying the effect where you want to selectively (ie. it's the concept of "mask selections", analgous to photoshop masks)

For example, you asked about RemoveDust() applying more speficially to white spots. Well one way to do this more selectively is to apply through a luma mask . Basically you generate a mask based on Y' values and that effect is applyed to the range that you specify. It's still not perfect because there will be overlaps (e.g. eye whites, white teeth, white lights etc....) . That's where the multiple inclusion / exclusion masks help (you limit , include or exclude by different parameters eg. you might want to limit by hue, or saturation, or edges etc...) . But the gold standard is still manual masking - nothing is more accurate than you manually identifying (this is why I use the strong/weak filtered layer technique)

[poisondeathray]

QTGMC won't get "rid" of the blending, it will probably introduce more blending (The "T" in QTGMC stands for temporal)

Between yadifmod+nnedi3 and QTGMC(), qtgmc worked better.

Hmmm define "better"

QTGMC is definitely making the blends worse, at least on a few frames

This section has more severly blended fields, and less blended fields. It's basically contaminating the less blended fields, making them more blended. It won't be possible to easily get a "perfect" result without a lot of manual work.

Some sections might look slightly better, but some are definitely worse. QTGMC applies a lot of post sharpening and some denoising by default, so people on cursory examination might think it looks better. It depends what your specific goals are, because each deinterlacer has pros/cons

[LaKap]

Thank you for the detailed explanation above.

QTGMC won't get "rid" of the blending, it will probably introduce more blending (The "T" in QTGMC stands for temporal)

Between yadifmod+nnedi3 and QTGMC(), qtgmc worked better.

Hmmm define "better"

QTGMC is definitely making the blends worse, at least on a few frames

This section has more severly blended fields, and less blended fields. It's basically contaminating the less blended fields, making them more blended. It won't be possible to easily get a "perfect" result without a lot of manual work.

Some sections might look slightly better, but some are definitely worse. QTGMC applies a lot of post sharpening and some denoising by default, so people on cursory examination might think it looks better. It depends what your specific goals are, because each deinterlacer has pros/cons

You are right. Pardon my ignorance. I do notice additional blending in less field blended sequences now.

I tried Vinverse() with Srestore() and few other things recommended on doom9 threads, but all gave similar unsatisfactory results. I am still figuring out what the "correct" or appropriate approach is for this. Please provide some pointers on approaching this with mvtools when you can.

[poisondeathray]

You are right. Pardon my ignorance. I do notice additional blending in less field blended sequences now.

I tried Vinverse() with Srestore() and few other things recommended on doom9 threads, but all gave similar unsatisfactory results. I am still figuring out what the "correct" or appropriate approach is for this. Please provide some pointers on approaching this with mvtools when you can.

Vinverse is for residual combing (there shouldn't be any after deinterlacing), and Srestore() partially depends on having access to some clean frames (so is partially dependent on the deinterlacer used)

Nothing will yield great results here, because every field has some clearly visible blending, at least in the 1st section. I want you to do a little exercise, examine it field by field and you will see blends everywhere

eg.
AssumeTFF()
SeparateFields()

The even fields are less severely blended in the 1st part with the man, so I would chose those when deinterlacing. But the odd fields are less blended in the 2nd part with the woman, so I would choose those. Odd fields are better in the 3rd part with the man. Any temporal deinterlacer will likely make the blends worse as with QTGMC (so I would stick with spatial)


I'm not sure what you're asking about with mvtools ? Can you clarify ?

[jagabo]

Please provide some pointers on approaching this with mvtools when you can.

You can try functions like RX() or my modified ReplaceFramesMC() and InsertFramesMC(). Those will replace (or insert) motion interpolated frames between other frames.

https://forum.videohelp.com/threads/352741-Frame-interpolation?p=2226119&viewfull=1#post2226119

I noticed that there were sections where portions of the frame bounced left and right alternating between frames. I thought that getting rid of every other frame then interpolating the missing frames would help:

SelectEven()
DoubleFPS2() #motion interpolated frame rate doubling

It did help, but the pattern wasn't consistent. It would go LRLRLR for a while but then would switch: LRLRLRRLRLRL. So there were still bounces, just fewer of them. You would get a pattern like: L_L_L_R_R_R. You could work around that with a lot of ReplaceFramesMC() but motion interpolation like this often doesn't work well.

I don't think it's practical to use compositing techniques and masks like poisondeathray suggested. Unless you plan to make fixing this movie this your life's work.

[poisondeathray]

I don't think it's practical to use compositing techniques and masks like poisondeathray suggested. Unless you plan to make fixing this movie this your life's work.

Exactly - it's a LOT of work, even if you're already familiar with the tools. Probably days/weeks/months just to even learn basic usage of the tools

Most people can't even be bothered to specify frame numbers for interpolation functions LOL

I'm always curious about these "Indian" movies, and how they got to that condition. It must be a race to make the most problems



LaKap - If you were asking about mvtools2 interpolation and the field blended section - you need "good" fields or frames to interpolate from . If your derive info from blended fields ,you propogate those blends. Step through the separated fields , and you will see many, many blends. It's basically useless for that section

[jagabo]

Here's a short example from the end of LK-sample2.avi using ReplaceFramesMC(). Original on the left interpolated on the right. At first it looks pretty good -- but look at the very bottom and very top of the frame.

[LaKap]

I want you to do a little exercise, examine it field by field and you will see blends everywhere

eg.
AssumeTFF()
SeparateFields()

I notice duplication when viewing each field separately. This has become more complicated that I had initially expected it to be .

LaKap - If you were asking about mvtools2 interpolation and the field blended section - you need "good" fields or frames to interpolate from . If your derive info from blended fields ,you propogate those blends. Step through the separated fields , and you will see many, many blends. It's basically useless for that section

I understand. What would be the your suggested course of action without getting in to the manual process of going frame by frame to fix it?

You can try functions like RX() or my modified ReplaceFramesMC() and InsertFramesMC(). Those will replace (or insert) motion interpolated frames between other frames.

https://forum.videohelp.com/threads/352741-Frame-interpolation?p=2226119&viewfull=1#post2226119

I noticed that there were sections where portions of the frame bounced left and right alternating between frames. I thought that getting rid of every other frame then interpolating the missing frames would help:

SelectEven()
DoubleFPS2() #motion interpolated frame rate doubling

It did help, but the pattern wasn't consistent. It would go LRLRLR for a while but then would switch: LRLRLRRLRLRL. So there were still bounces, just fewer of them. You would get a pattern like: L_L_L_R_R_R. You could work around that with a lot of ReplaceFramesMC() but motion interpolation like this often doesn't work well.

I don't think it's practical to use compositing techniques and masks like poisondeathray suggested. Unless you plan to make fixing this movie this your life's work.

I have not used RX or ReplaceFramesMC before, but I will try it out now. Thank you!

Here's a short example from the end of LK-sample2.avi using ReplaceFramesMC(). Original on the left interpolated on the right. At first it looks pretty good -- but look at the very bottom and very top of the frame.

Hmm... impressive, but yes, the artifacts on bottom and top are clearly noticeable. Would you mind sharing your script?

[poisondeathray]

jagabo - did you try adjusting the mvtools settings ? Maybe the vpad or VOverlap settings ? And/or try cropping the bottom border first ?

LaKap - If you were asking about mvtools2 interpolation and the field blended section - you need "good" fields or frames to interpolate from . If your derive info from blended fields ,you propogate those blends. Step through the separated fields , and you will see many, many blends. It's basically useless for that section

I understand. What would be the your suggested course of action without getting in to the manual process of going frame by frame to fix it?

You have to live with some blends in that section. Recall there are bad blends, and not as bad blends. Take the lesser of "evils" and use either selectodd() or selecteven() depending on which field is less blended from each scene, after applying a non temporal deinterlacer

[LaKap]

You have to live with some blends in that section. Recall there are bad blends, and not as bad blends. Take the lesser of "evils" and use either selectodd() or selecteven() depending on which field is less blended from each scene, after applying a non temporal deinterlacer

That makes sense. I am not aware of these non-temporal deinterlacer you have mentioned. I know the difference between spatial and temporal, but which particular set of AviSynth deinterlacers are we talking about?

EDIT

Found Spatial Bob deinterlacer.

[poisondeathray]

You have to live with some blends in that section. Recall there are bad blends, and not as bad blends. Take the lesser of "evils" and use either selectodd() or selecteven() depending on which field is less blended from each scene, after applying a non temporal deinterlacer

That makes sense. I am not aware of these non-temporal deinterlacer you have mentioned. I know the difference between spatial and temporal, but which particular set of AviSynth deinterlacers are we talking about?

Yadifmod+nnedi3 is spatial and will give better results in this section than qtgmc in terms of blends

Either use SelectOdd/Even, or configure yadifmod/nnedi3 to do it directly on each scene . You have to decide per scene -at least- , because even on that short sample, "even" were better with the 1st shot of the man, "odd" with the woman and the 2nd shot of the man

[LaKap]

Yadifmod+nnedi3 is spatial and will give better results in this section than qtgmc in terms of blends

Either use SelectOdd/Even, or configure yadifmod/nnedi3 to do it directly on each scene . You have to decide per scene -at least- , because even on that short sample, "even" were better with the 1st shot of the man, "odd" with the woman and the 2nd shot of the man

I tried Sbdeint(mode=1).SelectOdd/Even and liked the output. Then I tried Yadifmod+Nnedi3 and it looks even better because I now know what I exactly want, thanks to you and others who replied in this thread.

interp=nnedi3(nsize=2, nns=2)
yadifmod(mode=0, edeint=interp)

Thank you for all your inputs .

[jagabo]

Hmm... impressive, but yes, the artifacts on bottom and top are clearly noticeable. Would you mind sharing your script?

Yes, I got bored replacing individual frames (that bumped to the left) and resorted to one big replacement of 40 frames:

ReplaceFramesMC(75,1)
ReplaceFramesMC(77,1)
ReplaceFramesMC(79,1)
ReplaceFramesMC(81,1)
ReplaceFramesMC(83,1)
ReplaceFramesMC(86,1)
ReplaceFramesMC(88,40)

If you remove ReplaceFramesMC(88,40) line and continue replacing only 1 or two frames at a time there will be much less distortion (and less smooth motion because the camera doesn't move smoothly).

Note: ReplaceFramesMC(88,40) replaces frames 80 to 127 with motion interpolated between frames 79 and 128. When the motion interpolator can't make sense of the motion it resorts to blending. Hence the distortion at the top and bottom of the frame in this vertical tracking shot.