Removing noise and dirt from footage & sharpening using Hybrid

[Akuma786]

Hi, I'm trying to remove all the noise from movie and sharpen the footage ( see sample below) using Hybrid or even ffmpeg which ever is best. Can someone tell me what is the best method to do this please.

https://easyupload.io/t7020w

[davexnet]

You can clean up some of those small artifacts in Avisynth, but those large areas of film damage on the right
will take more work

[jagabo]

those large areas of film damage on the right will take more work

I went ahead and did it in AviSynth.

[Akuma786]

You can clean up some of those small artifacts in Avisynth, but those large areas of film damage on the right
will take more work

Thanks looks much better

[Akuma786]

I went ahead and did it in AviSynth.

Wow thats very impressive. Thanks for that. Can I ask you to share the script you used on this please.

Also can the fps be changed from 25 to 24fps and it be upscale to 1080p. I want to see how it would look with your script please if possible.

[jagabo]

Here's the script I used with the addition of halo reduction, more noise reduction, and upscaling.

ffVideoSource("Seven Steps Of Kung Fu 1979 SAMPLE.mp4") 

ColorYUV(off_y=-12, gamma_y=25)
ColorMatrix(mode="rec.601->rec.709")
BilinearResize(480, height)
FineDehalo(rx=2.5, ry=1.0)
RemoveDirtMC(50, false)
KNLMeansCL(d=1, a=2, h=3)
SMDegrain(thsad=300, tr=3, PreFilter=4)
Sharpen(0.3, 0.0)
MergeChroma(aWarpSharp2(depth=15))
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=696, fheight=height)
Sharpen(0.3)

ReplaceFramesRIFE(481,1)
ReplaceFramesRIFE(1028,1)
ReplaceFramesRIFE(1057,3)
ReplaceFramesRIFE(1088,1)
ReplaceFramesRIFE(1178,1)
ReplaceFramesRIFE(1207,2)
ReplaceFramesRIFE(1236,3)

nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=1280, fheight=632)
aWarpSharp2(depth=3)
Sharpen(0.30)

nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=1920, fheight=948)
aWarpSharp2(depth=3)
CAS(0.7)

You can find ReplaceFramesRIFE() here:
https://forum.videohelp.com/threads/410553-Jumping-Hesitating-Video?highlight=replacef...fe#post2699556

I stepped through the video frame by frame to find those with the big blotches and added the ReplaceFramesRIFE() calls. It will be tedious to do with the entire movie.

In the end, I'm not sure this is really an improvement. And it needs some work on the colors.

[Akuma786]

Thank you Jagabo. This looked very impressive. Epreciate all your efforts.

[Akuma786]

Here's the script I used with the addition of halo reduction, more noise reduction, and upscaling.

I've installed avisynth and when i open your script i get following error:

[Attachment 73659 - Click to enlarge]

I'm new to avisynth. what am i doing wrong would you know

[jagabo]

I believe mt_expand_multi is included in Dither Tools:
http://avisynth.nl/index.php/Dither_tools

Yes, there are hundreds of third part support filters. It can be hard to find everything you need.

[Akuma786]

Yes, there are hundreds of third part support filters. It can be hard to find everything you need.

Thanks Jagabo

Okay so I've managed to install the plugins required for your script so far. I'm now stuck on ReplaceFramesRIFE, I had a look on this forum for thread covering it but cant figure out what I'm doing wrong once again.

The errror message is :

[Attachment 73662 - Click to enlarge]

so I copied your code from (https://forum.videohelp.com/threads/410553-Jumping-Hesitating-Video?highlight=replacef...fe#post2699556) and saved as ReplaceFramesRIFE.avs into avisynth plugin folder.

Not sure this is what i am supposed to do.

here is the replaceFramesRife script i have saved:

[Attachment 73663 - Click to enlarge]

[jagabo]

Yes, copy/paste the code from that post and put it in an AVS script. Or paste it into your main script. But your ReplaceFramesRIFE.avs should contain only the two functions, ReplaceFramesRIFE and InsertFramesRIFE. Don't include the code after that.

ReplaceFramesRIFE.avs

function ReplaceFramesRIFE(clip Source, int N, int "X")
{
  # N is number of the 1st frame in Source that needs replacing. 
  # X is total number of frames to replace
  # e.g. RX(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for interpolation
  #
  # requires YV12, YV16, or YV24 input, outputs same as input

  X = default(X, 1) # default to replacing 1 frame

  Trim(Source, N-1, 1)+Trim(Source, N+X, 1)
  z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
  Rife(gpu_thread=1, model=9, factor_num=X+1, factor_den=1, sc=false)
#  Source.IsYV12 ? z_ConvertFormat(pixel_type="YV12", colorspace_op="rgb:709:709:f=>709:709:709:l") : last
#  Source.IsYV24 ? z_ConvertFormat(pixel_type="YV24", colorspace_op="rgb:709:709:f=>709:709:709:l") : \
#                z_ConvertFormat(pixel_type="YV16", colorspace_op="rgb:709:709:f=>709:709:709:l") 
   z_ConvertFormat(pixel_type=Source.pixeltype, colorspace_op="rgb:709:709:f=>709:709:709:l") 

  AssumeFPS(Source.FrameRateNumerator, Source.FrameRateDenominator)
  Trim(1,X)
#Subtitle("RIFE")

  Source.trim(0,N-1) ++ last ++ Source.trim(N+X,0)
}


function InsertFramesRIFE(clip Source, int N, int "X")
{
  # N is number of the 1st frame in Source that needs replacing. 
  # X is total number of frames to replace
  # e.g. RX(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for interpolation
  #
  # requires YV12, YV16, or YV24 input, outputs same as input

  X = default(x, 1)

  Trim(Source, N-1, 1)+Trim(Source, N, 1)
  z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
  Rife(gpu_thread=1, model=9, factor_num=X+1, factor_den=1, sc=false)
  z_ConvertFormat(pixel_type=Source.pixeltype, colorspace_op="rgb:709:709:f=>709:709:709:l") 

  AssumeFPS(Source.FrameRateNumerator, Source.FrameRateDenominator)
  Trim(1,X)
#Subtitle("RIFE")

  Source.trim(0,N-1) ++ last ++ Source.trim(N,0)
}

[jagabo]

Also can the fps be changed from 25 to 24fps

You can change to 24.0 fps with AssumeFPS(24.0). Or 23.976 fps with AssumeFPS(24000,1001). Those will also increase the running time by about 4 percent as they keep every frame and the frames are unchanged. If you're handling audio within the script add sync_audio=true to those commands.

I highly recommend against using DirectShowSource() as it's not frame accurate. It should be used a last resort when ffmpegSource and LSMASHSource filters have failed.

[Akuma786]

Yes, copy/paste the code from that post and put it in an AVS script. Or paste it into your main script. But your ReplaceFramesRIFE.avs should contain only the two functions, ReplaceFramesRIFE and InsertFramesRIFE. Don't include the code after that.

ReplaceFramesRIFE.avs

still getting error :


[Attachment 73667 - Click to enlarge]

this is the ReplaceFramesRIFE.avs which i have saved in the plugin64 folder of avisynth+ :

[Attachment 73668 - Click to enlarge]

Sorry to be a pain.

[jagabo]

You need to remove the bottom two lines of ReplaceFramesRIFE. Or include the entirety of InsertFramesRIFE. You might need it later anyway.

[Akuma786]

You need to remove the bottom two lines of ReplaceFramesRIFE. Or include the entirety of InsertFramesRIFE. You might need it later anyway.

Thank you Jagabo. I have finally managed to get the script to run on AvsPmod without any errors now!

so i opened the script on Virtualdub2 64bit and tried to save the file as mp4 and it crashes after a few minutes into rendering it.


[Attachment 73675 - Click to enlarge]

[jagabo]

Try disabling putting return(last) just before the first call to ReplaceFramesRIFE:

ffVideoSource("Seven Steps Of Kung Fu 1979 SAMPLE.mp4") 

ColorYUV(off_y=-12, gamma_y=25)
ColorMatrix(mode="rec.601->rec.709")
BilinearResize(480, height)
FineDehalo(rx=2.5, ry=1.0)
RemoveDirtMC(50, false)
KNLMeansCL(d=1, a=2, h=3)
SMDegrain(thsad=300, tr=3, PreFilter=4)
Sharpen(0.3, 0.0)
MergeChroma(aWarpSharp2(depth=15))
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=696, fheight=height)
Sharpen(0.3)

return(last)

ReplaceFramesRIFE(481,1)
ReplaceFramesRIFE(1028,1)
ReplaceFramesRIFE(1057,3)
ReplaceFramesRIFE(1088,1)
ReplaceFramesRIFE(1178,1)
ReplaceFramesRIFE(1207,2)
ReplaceFramesRIFE(1236,3)

nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=1280, fheight=632)
aWarpSharp2(depth=3)
Sharpen(0.30)

nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=1920, fheight=948)
aWarpSharp2(depth=3)
CAS(0.7)

That will essentially disable everything after the return(last). Can you render the result now? If not, try moving that return(last) earlier in the script and render again. Keep doing so until you find what is causing the problem.

There are many variations of RemoveDirtMC() so that's another suspicious filter. And I really don't trust DirectShowSource() -- try ffVideoSource instead.

[Akuma786]

Can you render the result now?

Worked after deleting removeDirt MC Line from script. But after that when i removed "return(last)" it started crashing again.

Basically the script wont work with removeDirt MC and these lines (after ReplaceFramesRife):

nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=1280, fheight=632)
aWarpSharp2(depth=3)
Sharpen(0.30)

nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=1920, fheight=948)
aWarpSharp2(depth=3)
CAS(0.7)

[jagabo]

So if you disable (put a # at the start of the line) or remove the RemoveDirtMC() line it still doesn't work? Make sure you have nnedi3 and aWarpSharp2 installed. And make sure you have any filters those filter rely upon. For example the RemoveDirtMC that I used requires fft3dfilter, mv_tools2, and RemoveGrain. And in turn, those filters may have some other requirements.

[Akuma786]

so I have installed all other filters required. Still crashes some times 1/3 into the movie so i managed to do it in 3 segments and then after joining it all together.

I would like toi know, is there way to not have the RIFE writing in the corner of video ?

also the audio is out of sync with DirectShowSource and when i change to ffVideoSource, there is no audio playing.

[jagabo]

Remove the Subtitle() line(s). They were there just so it was easy to see which frames were created by Rife.

ffVideoSource() only loads the video. If you're not changing the running time of the video there's no need to load the audio; you can simply mux with the original audio. If you need to edit the audio in AviSynth you can use ffAudioSource():

a = ffAudioSource("filename.ext") # get the audio
v = ffVideoSource("filename.ext") # get the video
AudioDub(v, a) # join them together for further processing

[Akuma786]

Thank jagabo. I think i understand, so this is what I've tried now:

a = ffAudioSource("D:\7 STEPS FILES\Seven Steps Of Kung Fu 1979 576P Dvd-Adc.mp4")
v = ffVideoSource("D:\7 STEPS FILES\Seven Steps Of Kung Fu 1979 576P Dvd-Adc.mp4")
AudioDub(v, a)
AssumeFPS(24.0)
sync_audio=true

The audio is now playing on AvsPmod but not in sync.

[jagabo]

sync_audio needs to inside the AssumeFPS():

a = ffAudioSource("D:\7 STEPS FILES\Seven Steps Of Kung Fu 1979 576P Dvd-Adc.mp4")
v = ffVideoSource("D:\7 STEPS FILES\Seven Steps Of Kung Fu 1979 576P Dvd-Adc.mp4")
AudioDub(v, a)
AssumeFPS(24.0, sync_audio=true)

[Akuma786]

sync_audio needs to inside the AssumeFPS():

Got it, Thank you so much for your assistance.

[Gelinox]

A little late... just saw your thread.
I know you're going to find it very bad, but only with a few DaVinci Resolve filters and a little bit of Neat.

Kung-Fu mod.mp4

[Akuma786]

Appreciate it Gelinox. I did use DaVinci Resolve for the final colour correction on some scenes.

[WxofxW]

Hello, and sorry for trying to revive an old thread. I would appreciate some help if possible. I'm trying to clean up some old files with visible static/noise at just about every scene change. I've checked other copies of the file and it's the same problem. So it looks like an issue during production.

I am attempting to use your script to replace those bad frames in one file. Your script mentions "ReplaceFramesRIFE(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for interpolation". However, my application of it only seems to replace the N variable of "101". The X variable doesn't appear to do anything with the other 4 frames I would like to remove. Could you please take a look at my scripts and see if there is something wrong within?

Here is my script, had to modify it because Hybrid AviSynth couldn't find the models:

[Attachment 84568 - Click to enlarge]

And here is the custom script I'm using within Hybrid:

[Attachment 84569 - Click to enlarge]

Any help would be much appreciated!

Thanks!

[jagabo]

RIFE won't work across scene changes. It won't find any motion vectors for interpolation.

[WxofxW]

Well that makes sense then. Thanks for the quick reply!

I'm very new to AviSynth and Hybrid, so I probably would have realized that if I'd had more experience.

I've tried running a number of different filters, and it does make the noise a lot less visible. But at the expense of way too much smoothing and loss of detail. I've verified that I can export the individual frames, clean up the damaged frames and compile the file again, but that will take a really long time to do for all the video files. It's a shame RIFE won't work. Deleting those bad frames and replacing the empty spaces with duplicates of the nearest good frames sounded like the best and quickest way to clean things up...

Do you know of any functions that do work across scene changes?

[poisondeathray]

Do you know of any functions that do work across scene changes?

DMVFN can be used to interpolate scene changes, but there is no vapoursynth/avisynth port yet.

Instead of interpolating between 2 reference frames, it uses 2 consecutive frames to predict the next frame.

https://github.com/megvii-research/CVPR2023-DMVFN

[WxofxW]

Thanks for that. I'll give it a look

Do you know of any functions that do work across scene changes?

DMVFN can be used to interpolate scene changes, but there is no vapoursynth/avisynth port yet.

Instead of interpolating between 2 reference frames, it uses 2 consecutive frames to predict the next frame.

https://github.com/megvii-research/CVPR2023-DMVFN