Automatically fixing VHS jittering (bad frames)

[Naltrex]

Hi! I've got a bit of a problem here:

I'd like to fix a case of bad frames in over 100 episodes of an 80s soap opera show. It was uploaded to YouTube by the producer of the show in its original status: no restoration whatsoever. I've attached an example file.

I know how to select and interpolate these frames manually, (only in VEGAS Pro; I'm a newbie to AviSynth and VirtualDub) but I can't do that for over 4000 minutes of footage obviously.

I'm seeking an automatic solution that can detect and correct these issues without manual input. It doesn't even have to interpolate the frames: just duplicating them would be enough.

Any ideas? I'm willing to pay for a programmer if needed to create a custom script.

[jagabo]

This fixes a lot of it:

##########################################################################
#
# replace every frame with a motion interpolated frame (between the frame before and the frame after)
#
##########################################################################

function InterpolateAll(clip c)
{
    e = SelectEven(c).InterFrame(FrameDouble=true, cores=4).SelectOdd()
    o = SelectOdd(c).InterFrame(FrameDouble=true, cores=4).SelectOdd()
    Interleave(e,o)
    Loop(2,0,0)
}

##########################################################################
#
# most of the bad frames are darker at the top left.  Here I look for frames that are darker (at the top left)
# than the frames before and after and replace them with a motion interpolated (from the frame
# before and the frame after) frame.
#
##########################################################################

function FixDarker(clip v)
{
    v
    Interp = InterpolateAll()#.Subtitle("interpolated")

    testclip = BilinearResize(4,4).PointResize(width,height).Crop(0,0,width/4, height/4)#.PointResize(width,height)
    tc1 = Subtract(testclip.Loop(2,0,0), testclip).mt_binarize(133) # darker than frame before?
    tc2 = Subtract(testclip.Trim(1,0), testclip).mt_binarize(133) # darker than frame after?
    testclip = Overlay(tc1, tc2, mode="multiply") # darker than both?

    ConditionalFilter(testclip, interp, last, "AverageLuma()", "greaterthan", "100")
}

##########################################################################


LWLibavVideoSource("Torre10.mp4", cache=false, prefer_hw=2) 
src = last

# first fix single bad frame
FixDarker()

# then two bad frames in a row
e = SelectEven().FixDarker()
o = SelectOdd().FixDarker()
Interleave(e, o)

StackHorizontal(src, last)

It screws up when a glitch is near a scene change (some scene change logic might help) and with longer glitches. In theory you could make this work with longer glitches by working with further separated frames (three in a row with SelectEvery(3,0), SelectEvery(3,1, SelectEvery(3,2)) but the motion interpolation and scene change issues will get worse.

There's also some chroma changes near the bad frames. You can probably get rid of much of that with some strong temporal chroma filtering.

[johnmeyer]

Here is a script I wrote which will either output a list of frames that don't match either of the adjacent frames or, optionally, it will replace the bad frame with one created using motion estimation:

Finding individual "bad" frames in video; save frame number; or repair

Set the script options to repair and it will automatically replace the bad frame with one interpolated from its neighbors. There are also settings so you can display the detection parameters in order to set the threshold correctly.

If you use the script correctly, it should give you almost perfect results.

P.S. Just to be clear, you want to use the version of the script that I posted later in that thread. Here is a direct link to that version of the script:

Final Version

[johnmeyer]

I looked some more at your video and much of the time you have glitches which persist for more than one frame. There is no easy way to fix these. However, I have a Vegas Pro script which you can assign to a key in Vegas that will replace the glitch with a duplicate of the previous frame. This will require that you navigate to each bad spot and then press the repair key, but the fix is pretty quick. You could use this to do further work on the show after you run my AVISynth script above.

Vegas Pro Script To Replace Bad Frame With Duplicate of Previous Frame

/** 
* PURPOSE OF THIS SCRIPT:
*
* Fix a bad video frame by deleting bad frame and replacing with previous frame.
*
* A video track must be selected. If an audio track is selected, nothing happens.
*
* To use, select the event that contains the bad frame. Position the cursor so you can
* see the bad frame in the preview window. Then, run the script.
* 
* Copyright © John Meyer 2006
* Written: June 23, 2006
*
**/ 

import System; 
import System.IO; 
import System.Windows.Forms; 
import Sony.Vegas; 


try { 

  //Global declarations
  var dStart : Double;
  var dLength : Double;
  var dCursor : Double;
  var trackEnum : Enumerator; 
  var evnt : VideoEvent;
  var evnt2 : VideoEvent;
  var one_frame = Timecode.FromFrames(1);

  var track = FindSelectedTrack();                // Use this function to find the first selected track.  

  if (track.IsVideo()) {                          // Proceed only if selected track is video track.
    var one_frame = Timecode.FromFrames(1);
    var eventEnum  = new Enumerator(track.Events);
    dCursor = Vegas.Cursor.ToMilliseconds();      // Remember the cursor position.

    //Go through each event on the track.
    while (!eventEnum.atEnd()) {
      evnt = (eventEnum.item());

      // Get the event's start and length timecode, in milliseconds.
      dStart = evnt.Start.ToMilliseconds();
      dLength = evnt.Length.ToMilliseconds();

//    If the cursor timecode is between the beginning and end of the 
//    event timecodes, then split the event, delete first frame from event to right of split.
//    Copy last frame from event to left of split and put into the one-frame gap.

      if ( (dCursor > dStart) && ( dCursor <= (dLength + dStart) ) ) {
        var patch_frame : VideoEvent =  VideoEvent(evnt.Copy(VideoTrack(track),Vegas.Cursor));

        // Make first frame of new event equal frame to left of cursor (for all takes)
        for (var i=patch_frame.Takes.Count - 1; i >= 0; i--) {
          patch_frame.Takes[i].Offset = patch_frame.Takes[i].Offset + Vegas.Cursor - one_frame - evnt.Start;  // Make change for ALL takes.
        } 

        patch_frame.Start = Vegas.Cursor;     // Move new event to right of cursor.
        patch_frame.Length = one_frame;       // Truncate event to only one frame in length.

        evnt.Split (Vegas.Cursor - evnt.Start);

        // Next several lines truncate start of NEXT event by one frame.
        eventEnum.moveNext();                 // Go to next event on this timeline.
        evnt2 = VideoEvent(eventEnum.item()); // Get next event.
        evnt2.AdjustStartLength(evnt2.Start+one_frame, evnt2.Length-one_frame, true);

        if (evnt.IsGrouped) {                 // Add new events to existing grouping.
          var grp : TrackEventGroup = evnt.Group;
          grp.Add(evnt2);
          grp.Add(patch_frame)
        }

      } // End if dCursor ...

      eventEnum.moveNext(); // Go to next event on this timeline.

    } // End While

  } // End If Track

  else {
    MessageBox.Show("You must select a video track.");
  }

} catch (e) { 
  MessageBox.Show(e); 
} 

// End of main program



// Beginning of functions

function FindSelectedTrack() : Track {
  trackEnum = new Enumerator(Vegas.Project.Tracks);
  while (!trackEnum.atEnd()) {
    var track : Track = Track(trackEnum.item());
    if (track.Selected) {
      return track;
    }
    trackEnum.moveNext();
  }
  return null;
}

[Naltrex]

This fixes a lot of it:

##########################################################################
#
# replace every frame with a motion interpolated frame (between the frame before and the frame after)
#
##########################################################################

function InterpolateAll(clip c)
{
    e = SelectEven(c).InterFrame(FrameDouble=true, cores=4).SelectOdd()
    o = SelectOdd(c).InterFrame(FrameDouble=true, cores=4).SelectOdd()
    Interleave(e,o)
    Loop(2,0,0)
}

##########################################################################
#
# most of the bad frames are darker at the top left.  Here I look for frames that are darker (at the top left)
# than the frames before and after and replace them with a motion interpolated (from the frame
# before and the frame after) frame.
#
##########################################################################

function FixDarker(clip v)
{
    v
    Interp = InterpolateAll()#.Subtitle("interpolated")

    testclip = BilinearResize(4,4).PointResize(width,height).Crop(0,0,width/4, height/4)#.PointResize(width,height)
    tc1 = Subtract(testclip.Loop(2,0,0), testclip).mt_binarize(133) # darker than frame before?
    tc2 = Subtract(testclip.Trim(1,0), testclip).mt_binarize(133) # darker than frame after?
    testclip = Overlay(tc1, tc2, mode="multiply") # darker than both?

    ConditionalFilter(testclip, interp, last, "AverageLuma()", "greaterthan", "100")
}

##########################################################################


LWLibavVideoSource("Torre10.mp4", cache=false, prefer_hw=2) 
src = last

# first fix single bad frame
FixDarker()

# then two bad frames in a row
e = SelectEven().FixDarker()
o = SelectOdd().FixDarker()
Interleave(e, o)

StackHorizontal(src, last)

It screws up when a glitch is near a scene change (some scene change logic might help) and with longer glitches. In theory you could make this work with longer glitches by working with further separated frames (three in a row with SelectEvery(3,0), SelectEvery(3,1, SelectEvery(3,2)) but the motion interpolation and scene change issues will get worse.

There's also some chroma changes near the bad frames. You can probably get rid of much of that with some strong temporal chroma filtering.

First of all, thank you so so much to you both. You don't know how happy I was to see I got help with this project!

I'll be using jagabo's script since I found it easy to use and it fixes just enough bad frames. Thank you johnmeyer for the scripts too, I'll definitely be using the VEGAS script in the future, it's gonna be useful for some other projects I have in mind outside of TV series.

After troubleshooting and installing and uninstalling lots of plugins, I got the script to work with VirtualDub.

I'm just facing what seems to be my last issue: I need the script to output the final repaired video instead of outputting both the damaged and repaired video. I don't have enough technical knowledge to modify the script myself, so, it would be really appreciated if you could help me with that

[manono]

I need the script to output the final repaired video instead of outputting both the damaged and repaired video. I don't have enough technical knowledge to modify the script myself, so, it would be really appreciated if you could help me with that

Remove or comment out the last line of the script:

#StackHorizontal(src, last)

[jagabo]

Yes, the StackHorizontal() was just there for easy comparison of the original and filtered clip.

I found another call to FixDarker() after Interleave(e, o) fixes a few more frames.

And if you want to fix some more frames add this after the other fixes:

ReplaceFramesMC(409,3)
ReplaceFramePrev(787)
ReplaceFrameNext(789)
ReplaceFrameNext(1073)
ReplaceFramesMC(1823,7)
ReplaceFramesMC(2017)
ReplaceFramesMC(3060,5)

But those fixes were located manually, stepping through the video manually, frame by frame. You're probably not going to do this for a hundred episodes. You can find those ReplaceFrame... scripts in these forums.

[Marcio.ciconne]

This fixes a lot of it:

##########################################################################
#
# replace every frame with a motion interpolated frame (between the frame before and the frame after)
#
##########################################################################

function InterpolateAll(clip c)
{
    e = SelectEven(c).InterFrame(FrameDouble=true, cores=4).SelectOdd()
    o = SelectOdd(c).InterFrame(FrameDouble=true, cores=4).SelectOdd()
    Interleave(e,o)
    Loop(2,0,0)
}

##########################################################################
#
# most of the bad frames are darker at the top left.  Here I look for frames that are darker (at the top left)
# than the frames before and after and replace them with a motion interpolated (from the frame
# before and the frame after) frame.
#
##########################################################################

function FixDarker(clip v)
{
    v
    Interp = InterpolateAll()#.Subtitle("interpolated")

    testclip = BilinearResize(4,4).PointResize(width,height).Crop(0,0,width/4, height/4)#.PointResize(width,height)
    tc1 = Subtract(testclip.Loop(2,0,0), testclip).mt_binarize(133) # darker than frame before?
    tc2 = Subtract(testclip.Trim(1,0), testclip).mt_binarize(133) # darker than frame after?
    testclip = Overlay(tc1, tc2, mode="multiply") # darker than both?

    ConditionalFilter(testclip, interp, last, "AverageLuma()", "greaterthan", "100")
}

##########################################################################


LWLibavVideoSource("Torre10.mp4", cache=false, prefer_hw=2) 
src = last

# first fix single bad frame
FixDarker()

# then two bad frames in a row
e = SelectEven().FixDarker()
o = SelectOdd().FixDarker()
Interleave(e, o)

StackHorizontal(src, last)

It screws up when a glitch is near a scene change (some scene change logic might help) and with longer glitches. In theory you could make this work with longer glitches by working with further separated frames (three in a row with SelectEvery(3,0), SelectEvery(3,1, SelectEvery(3,2)) but the motion interpolation and scene change issues will get worse.

There's also some chroma changes near the bad frames. You can probably get rid of much of that with some strong temporal chroma filtering.

I tried to use this function and it gave me this error, could you help me?