FFmpeg - How to delete frames repeated more than 1 time

[frenksisco]

Very complex question. How can I remove with ffmpeg any frame repeated more than 1 time?
I mean, repeated 0 times (no equal frames) = ok, do nothing, repeated 1 time (2 equal frames) = ok, do nothing, repeated 2,3,4.......1000 times = remove frames after first repeat (after 2 equal frames)

Example: if ffmpeg recognizes a 20 times repeated frame, which means 21 equal frames, it has to delete 19 frames because I want to keep at least 2 equal frames, which corresponds to one frame repetition

I don't want any temporal interpolation, I don't want to keep the same number of frames. I really want those frames to be deleted from the count. The reason is that I have a very bad webcam video full of repeated frames in unpredictable places and I want to maintain a maximum tollerance of duplicate frames that does not exceed 2 frames (2 equal frames).
Is that possible?
I only know the way to remove all duplicate frames with this script

ffmpeg -i input.mov -vf mpdecimate,setpts=N/FRAME_RATE/TB -c:v prores_ks -profile:v 3 output.mov

But it is not the right one for me because it removes all duplicate frames

[october262]

Very complex question. How can I remove with ffmpeg any frame repeated more than 1 time?
I mean, repeated 0 times (no equal frames) = ok, do nothing, repeated 1 time (2 equal frames) = ok, do nothing, repeated 2,3,4.......1000 times = remove frames after first repeat (after 2 equal frames)

Example: if ffmpeg recognizes a 20 times repeated frame, which means 21 equal frames, it has to delete 19 frames because I want to keep at least 2 equal frames, which corresponds to one frame repetition

I don't want any temporal interpolation, I don't want to keep the same number of frames. I really want those frames to be deleted from the count. The reason is that I have a very bad webcam video full of repeated frames in unpredictable places and I want to maintain a maximum tollerance of duplicate frames that does not exceed 2 frames (2 equal frames).
Is that possible?
I only know the way to remove all duplicate frames with this script

ffmpeg -i input.mov -vf mpdecimate,setpts=N/FRAME_RATE/TB -c:v prores_ks -profile:v 3 output.mov

But it is not the right one for me because it removes all duplicate frames

try this here - https://stackoverflow.com/questions/72483584/how-to-remove-a-frame-with-ffmpeg-without-re-encoding

[Cornucopia]

How do you know they are not supposed to be there? And do you know if your video is storing it efficiently anyway (so moot point)?


Scott

[frenksisco]

try this here - https://stackoverflow.com/questions/72483584/how-to-remove-a-frame-with-ffmpeg-without-re-encoding

I'm sorry but it doesn't seem useful to my case. Or maybe I didn't understand the analogy....

How do you know they are not supposed to be there? And do you know if your video is storing it efficiently anyway (so moot point)?


Scott

You're right, I didn't explain it. Because a 15 fps video can't be correct if it has 3, 5, 20, or even more than 80 consecutive equal frames... Anyway, I don't care about respecting the correct frame rate or speed of the video. For a variety of reasons I have calculated that I can accept, where it happens, a couple of equal frames but no more. This is why I am looking for an ffmpeg script that if it finds 2 equal frames will accept them, if it finds 3 equal frames will delete 1, if it finds 30 equal frames will delete 28, and so on..

[pandy]

You could give a chance to https://ffmpeg.org/ffmpeg-filters.html#thumbnail filter - for setting like 'thumbnail=2' it may effectively decimate your file by half removing at least some repeated frames - but your goal can be impossible to do without something more complex - perhaps exporting video to image and using more filters with script.

Something like:
https://pyimagesearch.com/2014/09/15/python-compare-two-images/
https://pyimagesearch.com/2017/06/19/image-difference-with-opencv-and-python/

[_Al_]

That is perhaps not that difficult at all to do in vapoursynth, analyze video first, check frame differences using a threshold and making a list of to be deleted indexes, then actually delete them. Im sure Avisynth could have similar approach or using two scripts.

[frenksisco]

I didn't understand, or maybe I don't have the knowledge to understand the "thumbnail" way, and I don't have a clue how to do that with avisynth or vapoursynth. I barely know the basic functions of ffmpeg. Anyway I will study these options, thank you

[pandy]

I didn't understand, or maybe I don't have the knowledge to understand the "thumbnail" way, and I don't have a clue how to do that with avisynth or vapoursynth. I barely know the basic functions of ffmpeg. Anyway I will study these options, thank you

example for ffplay so you can visually check how filter works - scale is to speed up analysis process - feel free to modify example

ffplay -i "%1" -an -vf "scale=256:-8:sws_flags=spline,thumbnail=2"

thumbnail description

Seem ffmpeg support also some 'deep neural networks based' filters https://ffmpeg.org/ffmpeg-filters.html#dnn_005fclassify and others

Simple video processing can be difficult if for example noise is present - no clue how your stream looks - that's why mentioning also NN.

[_Al_]

vapoursynth solution could be:

import vapoursynth as vs
from vapoursynth import core

THRESHOLD = 0.001
MAX_DUPLICATES = 1 #means 2 subsequent same frames allowed

clip = core.lsmas.LibavSMASHSource('video.mp4')
clip = core.std.PlaneStats(clip, clip[0]+clip)

to_be_deleted_indexes = []
duplicates = 0
print('analyzing clip frame differences, wait ...')
for n, frame in enumerate(clip.frames()):
    if frame.props['PlaneStatsDiff'] > THRESHOLD:
        duplicates = 0
        continue
    else:
        duplicates += 1
        if duplicates > MAX_DUPLICATES:
            to_be_deleted_indexes.append(n)
       
print('done analyzing')
print(f'number of frames to be deleted: {len(to_be_deleted_indexes)}')
#print(f'actual frame indexes to be deleted: {to_be_deleted_indexes}')

clip = clip.std.DeleteFrames(to_be_deleted_indexes)   
clip.set_output()

also, what to do with it, that script above could be named: "deleting_frames.vpy", then using:

"vspipe.exe" --outputindex 0 --container y4m "deleting_frames.vpy" - | "x264.exe" --demuxer y4m --crf 18  --vbv-bufsize 30000 --vbv-maxrate 28000 --colorprim bt709 --transfer bt709 --colormatrix bt709 --output deleted_frames.264 -

or

"vspipe.exe" --outputindex 0 --container y4m "deleting_frames.vpy" - | "ffmpeg.exe" -i - ....

to get avc stream that you can mux into some mp4 or mkv for a proper playback,
vspipe comes with vapoursynth, encoder (x264 in this case, but you can use ffmpeg also) you have to download
playback test with mpv player (limited seeking, because it is a raw stream):

vspipe --outputindex 0 --container y4m "deleting_frames.vpy" - | mpv -

or use some vapoursynth previewer for a visual feedback before encoding

[frenksisco]

I really thank you very much _Al_ I am sure it works but at the moment I don't understand it. As I said I don't have the minimum knowledge to understand what you are showing me. I don't know how to use vapoursynth, I don't know what vspipe is and I don't know where I have to write these scripts. That is why I was asking for something more familiar to me like an ffmpeg script. But as soon as I have time I will study this. Thanks again.

pandy your solution is a little little more understandable to me but I need to figure out how to merge it into my script

ffmpeg -i input.mov -c:v prores_ks -profile:v 3 output.mov

[pandy]

pandy your solution is a little little more understandable to me but I need to figure out how to merge it into my script

ffmpeg -i input.mov -c:v prores_ks -profile:v 3 output.mov

you can try something like:

ffmpeg -i input.mov -c:v prores_ks -profile:v 3 -vf "scale=960:-8:sws_flags=spline,thumbnail=2" output.mov

or without resize

ffmpeg -i input.mov -c:v prores_ks -profile:v 3 -vf "thumbnail=2" output.mov

But resize will speedup process so you can decide if this is good for you or not, also quite important - "thumbnail=2" decimate by half so you may need multiple iterations to eliminate duplicate frames - as such i would not use lossy codec (even tailored for video processing due eventual accumulation of the compression errors),
you can also stuck multiple "thumbnail=2" in series if you are sure that it will work for you so your script may look like this:

ffmpeg -i input.mov -c:v prores_ks -profile:v 3 -vf "thumbnail=2,thumbnail=2,thumbnail=2,thumbnail=2,thumbnail=2" output.mov

this is quite radical decimation.

[frenksisco]

But I'm not sure, I don't have to decimate my frames by half, I need to keep 2 of them only when and where it finds more than 2 equal.

p.s. ehm... I wanted to attach a video example to better explain what I need to do.... But I just found out that none of these buttons work.


So now I have a problem within a problem....

[_Al_]

Upload Image works,
"Add an Image" window would popup, select "From Computer" Tab, then "Browse" button, then "Upload Files" button

[frenksisco]

Yes Upload image works in "quick reply" mode, not in "go advance" mode. Really strange. Anyway, I was able to upload the video to at least paste the link
https://files.videohelp.com/u/182695/with-drops.mp4
As you can see it is a 50 fps video composed mainly of pairs of equal frames. (a 25fps transposed to 50 fps without interpolation) I added a few blocks of equal frames to it here and there (sometimes I added only 1, sometimes more than 10) and chroma reversed these added frames so that they would be more noticeable.
What I need is a script that removes any frame that is repeated more than 2 times. FFmpeg should recognize - where - there are more than 2 identical frames and remove them. So if the script works you should not see frames with inverted colors. Like this https://files.videohelp.com/u/182695/without-drops.mp4

[_Al_]

inverting a frame disables algorithms to detect a difference from original frame, if you want someone to test your video, better not to invert added frames

edit: Or perhaps you posted those videos just to demonstrate what you want, ok

[pandy]

As i wrote earlier - there is no ready to use filter in ffmpeg, also i doubt is there any ready to use filter as your problem is quite exceptional.
Probably you need to do multiple steps to get satisfactory results.
In theory you can use this: https://github.com/KenjiTakahashi/mpdecimate_trim but probably need to modify script as it has hardcoded some settings (compression codec and detection threshold).

[frenksisco]

inverting a frame disables algorithms to detect a difference from original frame, if you want someone to test your video, better not to invert added frames

edit: Or perhaps you posted those videos just to demonstrate what you want, ok

You are right, this is the link to the video without inverted chroma https://files.videohelp.com/u/182695/with-drops-no-inverted.mp4

As i wrote earlier - there is no ready to use filter in ffmpeg, also i doubt is there any ready to use filter as your problem is quite exceptional.
Probably you need to do multiple steps to get satisfactory results.
In theory you can use this: https://github.com/KenjiTakahashi/mpdecimate_trim but probably need to modify script as it has hardcoded some settings (compression codec and detection threshold).

Okay I understand, I will look at "mpdecimate_trim" thank you very much

[jagabo]

This sounds like an ill conceived request. But like this?

[_Al_]

posted vapoursynth script above, though with THRESHOLD=0.005 yielded:
number of frames to be deleted: 43
actual frame indexes to be deleted: [1, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 59, 72, 81, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 159, 160, 161, 162, 163, 164, 165, 166, 167]
It also caught second frame 1, second frame, it looks that one was added also. Frames 0 and 1 are not the same. Frames are always indexed from zero, so first frame has zero index.

[frenksisco]

This sounds like an ill conceived request. But like this?

No idea how you did it, but it is perfect

posted vapoursynth script above, though with THRESHOLD=0.005 yielded:
number of frames to be deleted: 43
actual frame indexes to be deleted: [1, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 59, 72, 81, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 159, 160, 161, 162, 163, 164, 165, 166, 167]
It also caught second frame 1, second frame, it looks that one was added also. Frames 0 and 1 are not the same. Frames are always indexed from zero, so first frame has zero index.

I am sure your method will work, but first I need to understand what Vapoursynth is and how to use it.

[jagabo]

This sounds like an ill conceived request. But like this?

No idea how you did it, but it is perfect

I did it with AviSynth using a filter called DeDup(). It requires two passes. In the first pass:

LWLibavVideoSource("with-drops-no-inverted.mp4") 

e = SelectEven()
o = SelectOdd()

e = e.DupMC(log="e.dup.txt")
o = o.DupMC(log="o.dup.txt")

Interleave(e, o)

builds a pair of status files about the video. Then in the second pass it removes duplicates over 2:

LWLibavVideoSource("with-drops-no-inverted.mp4") 

e = SelectEven()
o = SelectOdd()

e = e.DeDup(threshold=4.0, maxcopies=16, maxdrops=15, log="e.dup.txt", times="e.times.txt")
o = o.DeDup(threshold=4.0, maxcopies=16, maxdrops=15, log="o.dup.txt", times="o.times.txt")

Interleave(e, o)

You can open the first script in VirtualDub2 and select File -> Run Video Analysis Pass. That executes the entire script. Open the second script in VirtualDub2 (or whatever encoder you want that has AviSynth support) and encode. You may have to adjust the threshold values.

Keep in mind that the video will have fewer frames and hence a shorter running time -- so the original audio will not sync with it.

[frenksisco]

Just to make sure I explained it right. The video I am trying to fix is not "simple" like the video I uploaded. The video I uploaded is just a simplified example made of 25 double frames (after being fixed).... The real video I have to fix is not made of 25 double frames, it is a 15 fps buggy webcam video that frequently has a pair of equal frames, sometimes it has a block of 10 or 15 equal frames, sometimes 5 equal frames, sometimes 9, sometimes 3, sometimes 25 and so on... And as I said I can only accept 2 equal frames, if there is more they have to be eliminated. The optimum, of course, would be no double frames at all but if I eliminate them all the speed of the video becomes unnatural.

[jagabo]

The script I gave should be able to handle more irregular repeat patterns.

[frenksisco]

I consider the problem solved. The solutions you proposed work, now I just need some time to learn how to use them. Thank you all very much for your help