RIFE Not Interpolating During Fast Panning

[Alwyn]

I have a short piece of H264 video shot with my phone in UHD 30fps Variable. I have exported it to a 30fps Constant FR AVI for processing.

I started off with the phone virtually stationary, then panned, and then slowed down the pan to stop. What I've found is this:

Slow movements at the start have been interpolated nicely. But as the panning speed increases, the interpolation stops and I get motion every 2nd frame. Then, at the end, as I slow down the pan to almost stopped, the interpolation kicks in again.

Here's my code:

AVISource("F:\Fete\End pan 30fps.avi") 
ConvertToYV16

z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
Rife(gpu_thread=1, model=71, fps_num=60000, fps_den=1000, uhd=true, sc=true, sc_threshold=0.12)
z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:f=>709:709:709:l")

For privacy reasons I can't post the file but if necessary, will try to make another tomorrow to try to simulate this problem file.

I have tried various models but with no luck. I also tried Frame Rate Converter but it was throwing up blended frames.

I'd really appreciate any suggestions you could provide or comments; is this a characteristic of RIFE with UHD?

[Sharc]

The root cause may be the variable framerate (VFR) of mobile phone clips. Try to convert it to constant frame rate (CFR) before RIFE. Just speculating.

[Alwyn]

Thanks Sharc, already did that, as mentioned. I did it by saving in MagicYUV in VDub; I assume that will give me CFR?

[Sharc]

Thanks Sharc, already did that, as mentioned. I did it by saving in MagicYUV in VDub; I assume that will give me CFR?

Hmmm yes, how does MediaInfo report the framerate of the converted .avi? How does LWLibavVideoSource or some other source filter import it? Again, just guessing ....

Edit: Maybe the fast panning was interpreted as scene changes. Try sc=false, or increase the threshold.

[jagabo]

Yes, large motions between frames will be interpreted as scene changes. But even with sc=false motion vectors may not be found.

[Alwyn]

I've edited out the sensitive bits; here's the file. The interpolating starts as I slow down the pan at the lady with the scarf, stops as I reverse the pan and then starts again right at the end on the chair legs.

[Alwyn]

SC=false made a big difference and it comes out great now, just like the RIFE I know. A couple of minor hiccups here and there but much better. The grass is a bit messy but I'm not using that bit.

Thanks for your help guys.

[Sharc]

sc=false seems to work here (or set the sc_threshold=0.22 for example. You have to find the sweet spot for handling real scene changes).

Edit: glad it works for you too.

[rgr]

It's better to use sc1=True. Sc=True leaves duplicates at the scene change. Sc1=True blends the duplicate with the first frame of the new scene.

[rgr]

Thanks Sharc, already did that, as mentioned. I did it by saving in MagicYUV in VDub; I assume that will give me CFR?

You don't have to use VDub for this - LWLibavVideoSource without parameters will return the video in CRF just like VDub. BSsource too.

[Alwyn]

It's better to use sc1=True

You don't have to use VDub for this - LWLibavVideoSource without parameters will return the video in CRF just like VDub. BSsource too.

Thanks for both of those, rgr.

[rgr]

SC=false made a big difference and it comes out great now, just like the RIFE I know. A couple of minor hiccups here and there but much better. The grass is a bit messy but I'm not using that bit.

Thanks for your help guys.

Grass interpolation is highly model dependent - models 72, 41, 43, 45, 47, 52, 58 work best. In the attach, model 72.
However, I was looking for another file on the forum - does anyone remember the file from the rowing boats that had a big stuttering?

[DTL2023]

With update to ver 1.4.0 https://github.com/Asd-g/AviSynthPlus-RIFE it can now accept external scenechange signal. So users can make mvtools-based motion compensated script to produce sc_clip for RIFE and have same quality of scenechange detection as in mvtools. Non motion compensated scenechange detection ways can give more false positive signals at the high motion scenes.

Script is about

sup=MSuper()
mvs=MAnalyse(sup)
sc_c=MSCDetection(mvs) # adjust thSCD1 and thSCD2 if needed
RIFE(sc_clip=sc_c)

[lollo]

Excellent, thanks DTL2023!