RIFE: What do I need : How do I use it for interpolating frames?

[Charles-Roberts]

AviSynthPlus-RIFE
https://github.com/Asd-g/AviSynthPlus-RIFE/tree/1.2.0
https://github.com/Asd-g/AviSynthPlus-RIFE/releases/tag/1.2.1.1


Old release link for models.
https://github.com/Asd-g/AviSynthPlus-RIFE/releases/tag/1.1.0
https://github.com/Asd-g/AviSynthPlus-RIFE/releases/download/1.2.1/models.7z

There seems to be models update to 1.2.1?
Are they new models in 1.2.1 or just old models repackaged?
Keen to hear, learn more about the models.

New to this!
Whats needed for setup and are there code examples?

[jagabo]

I have the 1.1.0 version installed and running. You need the rife.dll file and the models. The models should be in the same folder as rife.dll (your plugins folder if you want it to autoload), in a folder called models:


[Attachment 77871 - Click to enlarge]

You can then call it with something like:

# assuming you video is already loaded
# Rife requires RGBPS color format, use z_convertformat, http://avisynth.nl/index.php/Avsresize

z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
Rife(gpu_thread=1, model=9, fps_num=60000, fps_den=1001, sc=true, sc_threshold=0.12)
z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:f=>709:709:709:l")

I tried updating to version 1.2 but couldn't get it to recognize the models folder.

[davexnet]

Does rife require a Nvidia gpu?

[Selur]

Yes, unless you use it through avs-mlrt (https://github.com/Asd-g/avs-mlrt) which does support different inference engines.
iirc avs-mlrt unlike vs-mlrt didn't come with RIFE support inside the main script, but it was possible to use it.
Others will hopefully be able to say more about this.

[rgr]

# assuming you video is already loaded
# Rife requires RGBPS color format, use z_convertformat, http://avisynth.nl/index.php/Avsresize

z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
Rife(gpu_thread=1, model=9, fps_num=60000, fps_den=1001, sc=true, sc_threshold=0.12)
z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:f=>709:709:709:l")

I tried updating to version 1.2 but couldn't get it to recognize the models folder.

It worked for me without any problems, but I downloaded new models simultaneously.
My comments:
1. Model 9 no longer supports custom frame rate ("Only rife-v4 model supports custom frame rate" --> models>=11(?)). Anyway still the best models are 5 and 6, sometimes 3 or 7.

2. With the new version, instead of sc=true, it is better to use sc1=true ("Blend frames (average) frames over scene changes.")



PS. The new Topaz intepolates frames in an interesting way:

Assuming that the numbers are existing frames and the numbers+letter are interpolated ones, it does it like this for doubling the frame rate:

1 1a 1b 2 2a 2b 3 3a 3b 4 4a (new scene)
1a 1b 2a 2b 3a 3b 4a (new scene)

where frame 4a is some interpolation of frame 4 and vectors from the previous frame

[Charles-Roberts]

1. Model 9 no longer supports ("Only rife-v4 model supports custom frame rate" --> models>=11(?)). Anyway still the best models are 5 and 6, sometimes 3 or 7.

I start with mpdecimate, the settings aggressively remove all duplicates.
Sometimes it will remove 1 or 2 original frames, but for low res/quality vids the end result after rife is arguably better for me.

C:\ffmpeg\bin\ffmpeg.exe -i C:\ffmpeg\bin\vid.mp4 -map_metadata -1 -vf mpdecimate=hi=64*30:lo=64*10:frac=1:max=0,setpts=1*PTS -c:v libx265 -an -x265-params lossless=1 -vsync vfr C:\ffmpeg\bin\vid-out.mp4 

Then I do any restoration processing.
Before rife using the multiply fps method.

DirectShowSource("vid.avi")
ConvertToPlanarRGB
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="rgb:709:709:f=>same:same:same:f")

#2xfps
Rife(gpu_thread=1, model=12, factor_num=200000, factor_den=100000, sc_threshold=1)
z_ConvertFormat(pixel_type="RGBP16", colorspace_op="auto:auto:auto:f=>same:same:same:f")

My questions:
1/Is my use, what they are referring to as custom frame rate?
Or can I use other models with my settings?

2/Not to argue about particular model, I have only used 11.
Could more people give opinion on the best model?

3/Can you somehow give locations for two ffmpeg.exe static builds to access both with single ffmpeg cmd/video process?
Perhaps merge the two static builds?
Any ideas?

I am generally interested out of curiosity, also where you could otherwise complete tasks in single process.
Only for my own personal use.

[buzz1891]

I start with mpdecimate, the settings aggressively remove all duplicates.

Do you mean regular (like 1-in-5) or completely random duplicates? Replacing the former is rather simple, but latter can be a challenge

[Charles-Roberts]

I mean "all" duplicates for the source used.

I put the unprocessed and processed side by side and compared over 100 frames.
Found 1 original removed, it was so similar that the end result with rife may have still been an improvement without it.

Even though frames are duplicated when people lift frame rate and should be identical in theory, these duplicates can develop differences.
My thinking is that other processes like reformat, attempted repairs or restoration could also create these differences.

The only difference could be brightness for example, and obviously variation in pixels to still be identified!
Yes, they were all removed from the particular source.

The only way to know is try, and you can expect every source to be different.

[Charles-Roberts]

What are they referring to as custom frame rate, because it seems that all frame rates are custom?
How can you use any other model other then 12?

I have tried countless ways such as saving 15fps as 30 or 29.97 and settings factor_num=1, factor_den=1,
The only model I have managed to get working in any form is 12.

How are people using other models like 5 and 6, can you share knowledge and example because I am out of ideas.

[poisondeathray]

What are they referring to as custom frame rate, because it seems that all frame rates are custom?
How can you use any other model other then 12?


How can you use any other model other then 12?

You can change the model by using model="x" , where "x" is any number that refers to a specific model

The models are listed in the readme

https://github.com/Asd-g/AviSynthPlus-RIFE

I have tried countless ways such as saving 15fps as 30 or 29.97 and settings factor_num=1, factor_den=1,
The only model I have managed to get working in any form is 12.

Only 4.x models can achieve any arbitrary framerate . Currently this model 11 or greater . Use fps_num, fps_den to achieve a set framerate. Factor_num, factor_den are multiplicative - so you can't achieve any frame rate.



Each model has pros/cons. There is no "best" for every source or even a single frame. One model might solve a frame perfectly , but fail with artifacts on another frame. But another model might solve that. You might need to use several models for different frames for near perfect results

[Charles-Roberts]

Yes I understand, but I would just like to see an example how any other models can be used for any purpose.
I have read everything and been at this for a week, my understanding is missing something!

Just a script or what ever it is called, using one of the other models so I can see how its used and identify what I am missing!

To reiterate, I have been using rife model 12 with no problem for both set frame rate and multiple of frame rate.
Obviously the other models are used some how!
It could be a simple mistake, I just don't know what!

[poisondeathray]

Just a script or what ever it is called, using one of the other models so I can identify what I am missing!

To use model 44 , to hit 29.97fps (30000/1001)

RIFE(model=44, fps_num=30000, fps_den=1001)

To use model 52, to hit 30/1 fps (30/1)

RIFE(model=52, fps_num=30, fps_den=1)

num and den are numerator and denominator , respectively.

[Charles-Roberts]

Just a script or what ever it is called, using one of the other models so I can identify what I am missing!

To use model 44 , to hit 29.97fps (30000/1001)

RIFE(model=44, fps_num=30000, fps_den=1001)

To use model 52, to hit 30/1 fps (30/1)

RIFE(model=52, fps_num=30, fps_den=1)

num and den are numerator and denominator , respectively.

I was trying the original format example I gave above, with values:
Rife(gpu_thread=1, model=5, fps_num=30000, fps_den=1001, sc_threshold=1)
Rife(gpu_thread=1, model=5, fps_num=30, fps_den=1, sc_threshold=1)

Seem to be the same as yours?
Need to double check it for errors!
I will copy this page and give it another go with your format.

Thank you

[poisondeathray]

model=5 won't work for fps_num , fps_den because it corresponds to rife 2.3 . Only 4.x models can hit any framerate. <4 can only do powers of 2, like 2x, 4x

The minimum model is 11 , which corresponds to 4

0: rife
1: rife-HD
2: rife-UHD
3: rife-anime
4: rife-v2
5: rife-v2.3
6: rife-v2.4
7: rife-v3.0
8: rife-v3.1
9: rife-v3.9_ensembleFalse_fastTrue
10: rife-v3.9_ensembleTrue_fastFalse
11: rife-v4_ensembleFalse_fastTrue
12: rife-v4_ensembleTrue_fastFalse
13: rife-v4.1_ensembleFalse_fastTrue
14: rife-v4.1_ensembleTrue_fastFalse
15: rife-v4.2_ensembleFalse_fastTrue
16: rife-v4.2_ensembleTrue_fastFalse
17: rife-v4.3_ensembleFalse_fastTrue
18: rife-v4.3_ensembleTrue_fastFalse
19: rife-v4.4_ensembleFalse_fastTrue
20: rife-v4.4_ensembleTrue_fastFalse
21: rife-v4.5_ensembleFalse
22: rife-v4.5_ensembleTrue
23: rife-v4.6_ensembleFalse
24: rife-v4.6_ensembleTrue
25: rife-v4.7_ensembleFalse
26: rife-v4.7_ensembleTrue
27: rife-v4.8_ensembleFalse
28: rife-v4.8_ensembleTrue
29: rife-v4.9_ensembleFalse
30: rife-v4.9_ensembleTrue
31: rife-v4.10_ensembleFalse
32: rife-v4.10_ensembleTrue
33: rife-v4.11_ensembleFalse
34: rife-v4.11_ensembleTrue
35: rife-v4.12_ensembleFalse
36: rife-v4.12_ensembleTrue
37: rife-v4.12_lite_ensembleFalse
38: rife-v4.12_lite_ensembleTrue
39: rife-v4.13_ensembleFalse
40: rife-v4.13_ensembleTrue
41: rife-v4.13_lite_ensembleFalse
42: rife-v4.13_lite_ensembleTrue
43: rife-v4.14_ensembleFalse
44: rife-v4.14_ensembleTrue
45: rife-v4.14_lite_ensembleFalse
46: rife-v4.14_lite_ensembleTrue
47: rife-v4.15_ensembleFalse
48: rife-v4.15_ensembleTrue
49: rife-v4.15_lite_ensembleFalse
50: rife-v4.15_lite_ensembleTrue
51: rife-v4.16_lite_ensembleFalse
52: rife-v4.16_lite_ensembleTrue
53: sudo_rife4_ensembleFalse_fastTrue
54: sudo_rife4_ensembleTrue_fastFalse
55: sudo_rife4_ensembleTrue_fastTrue
Default: 5.

[Charles-Roberts]

I also tried:
Rife(gpu_thread=1, model=5, factor_num=1, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2000, factor_den=1000, sc_threshold=1)

And other variation believing I understood this but the multiplier system did not work.
Just kept getting reply of only model 11-12 (4) for custom rate?

That's why I gave up after a week of lost hair and came here with question.
Possibly a simple typo or perhaps I am still missing something?

[poisondeathray]

I also tried:
Rife(gpu_thread=1, model=5, factor_num=1, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2000, factor_den=1000, sc_threshold=1)

And other variation believing I understood this but the multiplier system did not work.
That's why I gave up after a week of lost hair and came here with question.
Possibly a simple typo or perhaps I am still missing something?

factor_num=1, factor_den=1 should do nothing

You would expect factor_num=2, factor_den=1 to double the input frame rate

Another possibility is scene change threshold is wrong for your content. If frames are very different, everything might be detected as a scene change and interpolation is avoided everywhere

[Charles-Roberts]

I also tried:
Rife(gpu_thread=1, model=5, factor_num=1, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2000, factor_den=1000, sc_threshold=1)

And other variation believing I understood this but the multiplier system did not work.
That's why I gave up after a week of lost hair and came here with question.
Possibly a simple typo or perhaps I am still missing something?

factor_num=1, factor_den=1 should do nothing

You would expect factor_num=2, factor_den=1 to double the input frame rate

Another possibility is scene change threshold is wrong for your content. If frames are very different, everything might be detected as a scene change and interpolation is avoided everywhere

Regarding factor_num and factor_den, our understanding are the same.

I set sc_threshold=1 to totally avoid recognition of any scene change.
It worked perfectly to disable any scene change identification using model 12.
I am not interested in identifying scene changes or transitional effects.
Just working with clips.

I might be interpreting the sc_threshold documentation wrong?
I understand the point you are making, and will experiment with it!

[Alwyn]

You are aware all the model numbers and folders got changed recently? I got caught because the main RIFE wiki wasn't changed to reflect the model changes.

You need to use model 11 or higher. That will get you RIFE v4. Get the new models from here:

https://github.com/Asd-g/AviSynthPlus-RIFE/releases/tag/1.2.1

[poisondeathray]

To disable scene change completely , set sc=False - this is actually the default setting

Or there could be other issues - such as source filter (Directshowsource is not necessarily frame accurate, it's a "last resort" filter) , or timestamp issues (e.g. ffmpeg mpdecimate with setpts can sometimes cause issues)

Try on a different "normal" video, like a random youtube video

If you still can get it to work on your video, post a sample

[Charles-Roberts]

You are aware all the model numbers and folders got changed recently? I got caught because the main RIFE wiki wasn't changed to reflect the model changes.

You need to use model 11 or higher. That will get you RIFE v4. Get the new models from here:

https://github.com/Asd-g/AviSynthPlus-RIFE/releases/tag/1.2.1

Yes, am aware.
Chees

[Charles-Roberts]

To disable scene change completely , set sc=False - this is actually the default setting

Or there could be other issues - such as source filter (Directshowsource is not necessarily frame accurate, it's a "last resort" filter) , or timestamp issues (e.g. ffmpeg mpdecimate with setpts can sometimes cause issues)

Try on a different "normal" video, like a random youtube video

If you still can get it to work on your video, post a sample

I have only used Directshowsource and Avisource, any recommendation and would I need additional bits to use.

Struck some time stamp issues, but not the issue in this case.
Also Tried different videos
More to look at, thanks.

[poisondeathray]

For source filters you can use ffms2 or l-smash
https://github.com/FFMS/ffms2/releases
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

FFVideoSource("video.mp4")

For L-smash, if you have timestamp issues from ffmpeg setpts, use LWLibavVideoSource instead of LSmashVideoSource, because the former will index the file. For a "normal" mp4 video, the latter is generally preferred because of no indexing time

LWLibavVideoSource("video.mp4")

Also Tried different videos

Did RIFE not work on the other videos ?

[Charles-Roberts]

That has given me a lot more knowledge and things to try.
Thanks for your help

[Charles-Roberts]

For source filters you can use ffms2 or l-smash
https://github.com/FFMS/ffms2/releases
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

FFVideoSource("video.mp4")

For L-smash, if you have timestamp issues from ffmpeg setpts, use LWLibavVideoSource instead of LSmashVideoSource, because the former will index the file. For a "normal" mp4 video, the latter is generally preferred because of no indexing time

LWLibavVideoSource("video.mp4")

Also Tried different videos

Did RIFE not work on the other videos ?

Rife model 11-12 (4) worked perfectly on every video, just not with the other models.
The results with 12 were very impressive.
I will try the suggestions we talked about.
There is a lot here I was not aware of.

[rgr]

I also tried:
Rife(gpu_thread=1, model=5, factor_num=1, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2, factor_den=1, sc_threshold=1)

Rife(gpu_thread=1, model=5, factor_num=2000, factor_den=1000, sc_threshold=1)

And other variation believing I understood this but the multiplier system did not work.
Just kept getting reply of only model 11-12 (4) for custom rate?

That's why I gave up after a week of lost hair and came here with question.
Possibly a simple typo or perhaps I am still missing something?

With model 5 you can only double the frame rate (from 30 to 60, from 29.97 to 59.94, etc.). You won't make 60fps from 24fps, for example.

Also use sc1=true:
"sc1
Blend frames (average) frames over scene changes.
This cannot be true when sc=true.
Default: False."