ReplaceFrameX & InsertFrameX

[hello_hello]

ReplaceFrameX is an Avisynth function for replacing bad frames. It can't look for bad frames itself, only replace those you tell it to. Frames can be replaced with duplicates, blended versions of the surrounding frames, or by interpolating frames using FrameRateConverter or MVTools2. The default is to replace frames with duplicates.

InsertFrameX is similar to ReplaceFrameX, but new frames are inserted into the video increasing the frame count (ie to add missing frames). There's an additional argument for inserting silent audio to match, although by default any audio is passed through untouched.

StartBlack, EndBlack, ReplaceBlack, AddBlack & AppendBlack are functions for adding black frames.

Usage instructions at the top of each script.

Edit 2023-01-07 Re-uploaded ReplaceFrameX to fix an error when the number of frames to be replaced wasn't specified. The version date remains the same.
ReplaceFrameX 2022-12-17.avsi
InsertFrameX 2022-12-17.avsi
Black (Start, End, Replace, Add, Append) 2022-10-16.avsi

Some ReplaceFrameX "Blend" and "FRC" examples.

ReplaceFrameX(56488, Mode="Blend", Position=true) - Replacing frame 56488



ReplaceFrameX(56488, Mode="FRC", Position=true) - Replacing frame 56488



ReplaceFrameX(56488, 2, "Blend", Position=true) - Replacing frames 56488 & 56489



ReplaceFrameX(56488, 2, "FRC", Position=true) - Replacing frames 56488 & 56489



ReplaceFrameX(56488, 3, "Blend", Position=true) - Replacing frames 56488, 56489 & 56490



ReplaceFrameX(56488, 3, "FRC", Position=true) - Replacing frames 56488, 56489 & 56490

[hello_hello]

I just realised the handling of the first couple, and possibly last couple of frames is bad (otherwise it's okay). I'll upload a new version dated 2022-10-16 sometime soon(ish).

[hello_hello]

There's a link for the fixed version dated 2022-10-16 in the opening post.

To atone for my silliness, I've also added a link for some functions for replacing frames with black ones, or appending black frames to the beginning and/or end of the video.

[rgr]

Nice!

[hello_hello]

I've added a link for a function called InsertFrameX to the opening post. It's quite similar to ReplaceFrameX, but instead of replacing existing frames it inserts new ones.

There's also a link for a new version of ReplaceFrameX dated 2022-12-05. It's functionality hasn't changed but I fixed an error message relating to "out of range" frame numbers.

[jagabo]

RX(), ReplaceFramesMC(), InsertFramesMC(): https://forum.videohelp.com/threads/352741-Frame-interpolation

[hello_hello]

I had no idea those functions existed so I suppose I've re-invented the wheel. My functions came from helping a forum member via PM and I thought someone else might find them useful.

I like manono's idea for inserting frames with the ReplaceFramesMC function.
Loop(2,1325,1325)
ReplaceFramesMC(1326,1)
It's not obvious (at least to me) but quite simple.

If you don't mind I'll add MBlockFPS and MFlowFPS from your ReplaceFramesMC functions as options for ReplaceFrameX too.

Something like
ReplaceFrameX(56488, Replace=2, Mode="MBlock")

I compared MVTools2 with FrameRateConverter again and sometimes the MVTools2 frames look better and sometimes not, depending on picture content and resolution etc. Of course block size etc can change the result for both quite a bit, but it can't hurt to include MVTools2 as an option too.

[jagabo]

I also have a version that uses SVPFlow.

[poisondeathray]

I would consider adding RIFE versions options for interpolation, since it has finally made it to avisynth now . The avisynth version uses NCNN - so it can use basically any GPU that supports Vulkan , not just NVidia.

If you haven't used it yet - RIFE generally produces better results than MVTools2, especially with object occlusions. I posted several comparisons in the past

I translated the vapoursynth versions that I posted earlier at doom9 to AVS versions here, feel free do modify/add/improve.
https://forum.videohelp.com/threads/407773-Help-with-8mm-frame-rate-conversion#post2674302

If you're wondering why there are separate RXr1, RXr2... etc.. in addition to the RXr generic function, it's because RIFE 4.x models are faster and can achieve any non integer multiple, but the quality tends to be lower. So to use <4.x models, a separate downconversion step is required

I hardcoded it for YUV input, but one way it might be improved is to use isRGB, isY

[hello_hello]

I've been distracted by the real world lately, so I haven't played with RIFE interpolation yet, but I have added support for Dogway's two flavours of FrameRateConverter, along with new RIFE presets. I've also added MVTools2 modes based on jagabo's scripts. New links in the original post.

Mode = "Dup", "DupP", "DupN", "Blend", "FRC", "FRCEX", "FRCMIX", "MFlow", "MBlock"
Means
Duplicate Previous/Next, Blend, FrameRateConverter, FrameRateConverterEX, FrameRateConverterMIX, MFlowFPS or MBlockFPS (MVTools2).

Preset = "Slowest", "Slower", "Slow", "Normal", "Fast", "Faster", "Anime", "Rife", "RifeHQ", "RifeAnime".

There's updated instructions at the top of each script.

[rgr]

Great plugin! MBlock with one frame works effectively, but with two, the effectiveness is usually about 50%. RIFE is missing

[jagabo]

Here's a little hack to get RIFE working with my old ReplaceFramesMC:

function ReplaceFramesRIFE(clip Source, int N, int "X")
{
  # N is number of the 1st frame in Source that needs replacing. 
  # X is total number of frames to replace
  # e.g. RX(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for interpolation
  #
  # requires YV12, YV16, or YV24 input, outputs same as input

  Trim(Source, N-1, 1)+Trim(Source, N+X, 1)
  z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
  Rife(gpu_thread=1, model=9, factor_num=X+1, factor_den=1, sc=false) # model 9 needed for custom factor/rate
  Source.IsYV12 ? z_ConvertFormat(pixel_type="YV12", colorspace_op="rgb:709:709:f=>709:709:709:l") : last
  Source.IsYV24 ? z_ConvertFormat(pixel_type="YV24", colorspace_op="rgb:709:709:f=>709:709:709:l") : \
                z_ConvertFormat(pixel_type="YV16", colorspace_op="rgb:709:709:f=>709:709:709:l") 
  AssumeFPS(Source.FrameRateNumerator, Source.FrameRateDenominator)
  Trim(1,X)
#Subtitle("RIFE") # optional label to verify replacement

  Source.trim(0,N-1) ++ last ++ Source.trim(N+X,0)
}

As implemented it only supports 8 bit YV12, YV16, and YV24. You an easily modify it for other color formats/depths.

[rgr]

Thanks, that's the script I was thinking of

Is there any difference between z_ConvertFormat and traditional convertbits and converttoplanarRGB(with matrix)?

Can this script be included as an integral part of AviSynth so as not to add to every script?

[jagabo]

Is there any difference between z_ConvertFormat and traditional convertbits and converttoplanarRGB(with matrix)?

Using z_ConvertFormat is a little faster (though it hardly matters given how slow RIFE is) but the end result is the same (aside from any rounding error differences).

Can this script be included as an integral part of AviSynth so as not to add to every script?

As with any AviSynth function you can put it in an AVSI file in your plugins folder and it will autoload every time you start AviSynth.

[poisondeathray]

You could use "pixel_type=Source.PixelType" , to convert back to source pixel type , that would support higher bitdepths, other pixel types for sources

[jagabo]

Thanks pdr. So it becomes the less unwieldy

function ReplaceFramesRIFE(clip Source, int N, int "X")
{
  # N is number of the 1st frame in Source that needs replacing. 
  # X is total number of frames to replace
  # e.g. RX(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for interpolation
  #
  # requires YV12, YV16, or YV24 input, outputs same as input

  Trim(Source, N-1, 1)+Trim(Source, N+X, 1)
  z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
  Rife(gpu_thread=1, model=9, factor_num=X+1, factor_den=1, sc=false) # model 9 needed for custom factor/rate
  z_ConvertFormat(pixel_type=Source.PixelType, colorspace_op="rgb:709:709:f=>709:709:709:l") 
  AssumeFPS(Source.FrameRateNumerator, Source.FrameRateDenominator)
  Trim(1,X)
#Subtitle("RIFE") # optional label to verify replacement

  Source.trim(0,N-1) ++ last ++ Source.trim(N+X,0)
}

[rgr]

I don't know how AviSynth works internally, so I have a question.

If, for example, I have 20 frame swaps in the script, will the YV24->RGB->YV24 conversion occur only for these swapped frames?

[poisondeathray]

I don't know how AviSynth works internally, so I have a question.

If, for example, I have 20 frame swaps in the script, will the YV24->RGB->YV24 conversion occur only for these swapped frames?

Yes, because of the Trim() calls in that function . Only the replaced frames are affected. The other frames are copied as unaltered input uncompressed data

YV24->RGBPS (float)->YV24 such as in that function is a lossless roundtrip transform (I mean if you didn't apply something else like RIFE in the RGBPS step, just referring to the pixel format conversions). A YV12 source would not be, unless the chroma up/down sampling was changed to "point" resampling (nearest neighbor)

[rgr]

Thx.

Not to spam this thread:

'ReplaceFramesRIFE & "System exception - Access violation"'

https://forum.videohelp.com/threads/410735-ReplaceFramesRIFE-System-exception-Access-v...on#post2700891

[Genik]

Please help me find any way to insert (*,1,"Blend") the selected 128 frames from sequence consisting of 2996 frames.
This sequence is repeated more than 30 times, in total there are more than 110,000 frames in the project.

[jagabo]

So you want to insert 128 individual frames with a blend of the frame before and the frame after? All of those frames are within a sequence of 2996 frames somewhere in the middle of 110000 frame? The where the sequence is, and the total number of frames don't really matter. If I have this right I can put together a quick function for you.

Note that inserting a frame means the frame number of every frame after the insertion is increased by one. So it's easiest to insert frames starting at the end and working back to the beginning. For example, you have a 10 frame video, frames 0 to 9. And you've determined beforehand that you need to insert a frame at 3 and 6. If you insert a frame at 3 first, then you need to insert the next frame at 7 rather than 6. If you work backward you can insert a frame at 6, then insert at 3. In both cases you'll have a total of 12 frames when you're done. If you're replacing frames the frame numbers don't change so this is not a problem.

[Genik]

There is a project (more than 110,000 frames) that was completely transferred from pal to ntsc by removing one random frame from 23-24 frames. The missing frames, of course, ruined the entire video. After examining over 4,000 frames, I discovered a repeating sequence of 2,996 frames to which the deleted 128 frames needed to be added.
SelectEvery(2996, .....) allows to insert duplicate frames next to the deleted one in the entire project (about 36 sequences of 2996 frames each, 128x36=4608 new frames), but this solution also does not look very good.
Looks great if add a blended or interpolated frame using InsertFrameX(24, 1, "Blend"). I have a list with frame numbers (for one sequence of 2996 frames) before which I need to add a new blended frame. All that remains is to combine this list with InsertFrameX

[jagabo]

The way I usually deal with this is to double the frame rate (with interpolation, blending, whatever) then use SelectEvery(). After doubling the frame rate, every other frame is one of the original frames, between each of them is a motion interpolated frame. For example, in a sequence of frames where frame #4 is missing:

0123567

becomes:

0_1_2_3_5_6_7

The original frames are now at position 0, 2, 4, 6... and the underscores indicate an interpolated frame between them. You don't want to keep most of the interpolated frames but the one between 6 and 8 is the missing #4 of the original clip. So:

SelectEvery(N, 0 2, 4, 6, 7, 8, 10, 12, 14...)

gets all the original frames plus the desired interpolated frame (7 in the list).

[Genik]

That's what I was planning to do unless I found an easier way than writing a 3000 character sequence in SelectEvery(). Was hoping to find an elegant solution to just specify 128 frames from the sequence to use the lovely InsertFrameX script.

[jagabo]

You probably don't need a 3000 character sequence. In frame rate conversion like this (PAL to NTSC) you should see a pattern that repeats every 24 frames. After about every 30 seconds the pattern will shift by a frame to make up the difference between 24 fps and 23.976 fps.

[Genik]

Unfortunately this sequence is not standard. Through experimentation, I used your InsertFramesMC script to add interpolated frames. It can process almost 400 frames at a time, while InsertFrameX can process no more than 50 because it consumes a lot of memory. I split the project into 6-minute clips (about 9000 frame) and decided that it would be easier for me than trying to do everything at once.