Duplicates Frames. Cause and means?

[nji9nji9]

Hello! ... for the first time.

In some older family feast movies (from about the 90ies),
taken with unknown hardware I spotted that there are duplicate frames.
As I need to analyse the movies closely and framewise
for further algorithmic processing these dupes are very disruptive.
I had a closer look at the distribution of the dupes,
and it shows it is not arbitrarily, but in a kind of "pattern".
But not as simple as "every 5th frame a dupe" or so,
but mostly "8 times every 6th frame a dupe, then once the 7th,
then 8 times every 6th again" and so on.
(The values of 8, 6, 7 are different from movie to movie of course.
So are the frame rates).

So my question:
What happened there?
Are these really inserted duplicates?
(So should be removed)
Or are they "placeholders" of frames the camera couldn't deliver
for some reason?
(Then they must not be removed, because the timeline would be out of order.
But the dupes could be exchanged by time-wise intermediate frames,
generated from their neighbours).
I was not able to decide which of both is the case.
And - depending on the answer - how to do the job?

I was pointed to IVTC ... but that is not the cause
(if I get it right).

Acutally I'm the opposite of a prof, and do with VirtualDub2 mostly.

Greetings!

[manono]

It's often done as a way to bring up the framerate from a lower initial framerate. Like maybe if the source is 8 or 16mm film. They can usually be returned to the original framerate.

For a more informed response, you'll need to post 10 seconds or so directly from the source. No reencoding, no YouTube garbage. A section with steady movement.

[nji9nji9]

Maybe a misunderstanding:
I'm quite sure the source is digital too.
Maybe it's the source.
What kind of means exist to return to orginal framerate? (As the dupe frames are so complex distributed).

[manono]

Well, you did say 'movies'. and you did say '90's', and you did say 'camera'. I doubt they originated as digital although I was assuming they've been digitized since then. Otherwise I highly doubt they would have duplicate frames. No misunderstanding.

How to return them to the original framerate? AviSynth's TIVTC probably. But, unless and until you provide a sample no further help can be provided.

I was pointed to IVTC ... but that is not the cause
(if I get it right).

You don't. They didn't go through a traditional telecine, but a decimation filter (part of an IVTC) can probably remove the duplicate frames, more or less perfectly.

[johnmeyer]

You say "movies," but you say they were taken in the 1990s. Do you really mean that they are movies, shot on film and then transferred to video, or are they actually videos, shot on some unknown hardware?

If these are transfers from actual movie film, then there is one set of solutions to consider which involves removing duplicates that were added to make the movie play at the correct speed when transferred to video. However, if they are actual videotapes (or discs), then the problem is more likely something to do with faulty capture hardware, and most likely you will find that, in addition to the duplicates, there are probably corresponding dropped frames. That is a more difficult problem to solve, but a few years ago I did take a shot at it and developed an AVISynth script that can usually make the video look a little better.

[edit]I took a long time finishing my reply and see that there were two post since I started.

If the source was video, not film, the most likely problem is that they were analog videos which were then digitized. When the capture hardware is not set up correctly, it will drop frames. Then, to keep the video and audio in sync, it will duplicate frames a few frames later. I created a script that attempted to fix it. Here is a link to the discussion where I eventually posted my AVISynth script

Automatically fix dups followed (eventually) by drops

The script in post #6 is the one you should look at.

[nji9nji9]

First of all:
Thank you for your support!
I do have to apologize if I used the terms inconsistently and confusing.
I am from the "algorithmic side", and know not much about filming etc.

OK.
I'm trying to tie it up. More clearly this time I hope:

Recently I stumbled upon some digital movies that are (estimated from
the content) from the late 90ies, maybe a few years later.
There is no way for me to find out, which devices have been used originally.

The format of the movies now is avc1/mp4 and even wmv7.

Having a frame-by-frame look, it turned out that there are dupes.

These dupes are very disturbing for my further algorithmic analysis.
Beside they are artifical of cause, in sense of not showing real world moving.

So I have to do ... something.

I tried to find out,
if these dupes are simply inserted frames (for the fun of it )
- so they could simply be removed -
or if the dupes are at timestamps where (something) wasn't quick enough,
so a dupe was inserted instead of the correct frame
- so they must not be removed, but be exchanged by an interpolated frame.

But I was not successful in deciding the question
(visually, and by lenghts of motion vectors).

Also there is the problem of determing the positions of the dupe frames.

It became clear to me that I have to understand (roughly) the cause.

[edit] And there is another reason why it is important to understand "the pattern":
I have to distinguish the duplicated/ dropped frames from "real world" non-motion frames.

The IVTC filter
internal to VD2 (parameters: "Reduce frame rate/ Auto / Adaptive)
didn't do well for my material, as it seems to do a FIX reducing of
the frame rate by 1.25. Which means to "drop" every 5th frame in average.
But the movies I'm talking about do have various occurrence of dupes:
Every 6th. Another every 5th. And still another every 2nd.
(In addition the "deviations" I described above).
Doing the IVTC filter on the last one (every 2nd a dupe) for example
makes out of the 29 fps movie a 29/1.25 = 23.2 fps with still dupes in it.
While the correct solution would be to remove every second frame
to make it a 29/2=14.5.
If I do the last - by a simple "decimating" filter, everything is OK.
(Although not perfect still, as the AABBAABB-pattern in the original movie
"flips" sometimes AABBABBAA).
In that trivial case.
No chance of repairing in the other cases.

Up to now I was successful in not opening that other new tin AVISynth
And if I have a look at the scripts and the discussion about it ...

I had hoped of a ready-made filter that fixes a (maybe well-known) problem.

Well, you find me confused.

[manono]

Still no sample. Just a bunch of useless and meaningless words.

[nji9nji9]

@johnmeyer:

As far as I understand film can be ruled out as original source.

What was likely, around 2000 in consumer market:
Analog video or already digital devices?

I suspect the original source was analog video.

Dupes probably weren't generated by them?
So the reasons for the dupes must have been the digitizing.

If I get you right, then doing that in a bad manner
will result in duplicate AND in (corresponding) dropped frames?

"Dropped frames" are new to me.
As far as I understand, they show in "jumps"?
Viewed best at evenly movements.
Having a closer look there are evenly movements, and dupes, but no jumps.
What does this point to?

Still there remains the problem of algorithmic distinguishing true duplicate
from non-motion frames.

Anything wrong with my humble conclusions?

And what to do now?

Greetings.

[poisondeathray]

I suspect the original source was analog video.

Dupes probably weren't generated by them?
So the reasons for the dupes must have been the digitizing.

If it was analog video, but a "movie", then at least some of them were caused by the format - if they were NTSC land . Telecine 23.976 => 29.97 . Someone just blindly doing a deinterlace will leave duplicates, degrade the footage

If it was home video, or theatrical movie - it could have been edited and placed on different timeline settings , this causes format and framerate conversions over generations . Look for cuts and edits. In the "movie" case hard telecine edits (still in 29.97 fields), without removing pulldown can interrupt the cadence. This causes variable duplicates

There can be many causes and explantions

"Dropped frames" are new to me.
As far as I understand, they show in "jumps"?
Viewed best at evenly movements.
Having a closer look there are evenly movements, and dupes, but no jumps.
What does this point to?

If a recording device drops a frame, often it will manifest actually as a duplicate frame. A "placeholder" frame to keep sync

Lossy encoding can make that duplicate appear like a non duplicate. 2nd duplicate pair might manifest a slight motion due to the motion vectors on the later generation encode . For scripts you can often adjust threshold values to include/exclude scenarios

Depending on how you are analyzing the videos, the duplicates might not actually be there in the video bitstream. It might be "VFR" timecode video, and the duplicates introduced by your playback/viewing system

And what to do now?

If you want experienced people to help you out, post untouched samples

Instead of going through every single scenario that might have happened , a video is wroth 10,000 words.

[jagabo]

As far as I understand film can be ruled out as original source.

Why? A digital capture of a movie broadcast on TV is ultiimatly a film source.

What was likely, around 2000 in consumer market:
Analog video or already digital devices?

Both.

I suspect the original source was analog video.

Dupes probably weren't generated by them?

Not likely.

So the reasons for the dupes must have been the digitizing.

If I get you right, then doing that in a bad manner
will result in duplicate AND in (corresponding) dropped frames?

Dups AND/OR drops.

"Dropped frames" are new to me.
As far as I understand, they show in "jumps"?
Viewed best at evenly movements.

Yes.

Still there remains the problem of algorithmic distinguishing true duplicate
from non-motion frames.

It's pretty easy to compare consecutive frames and discard duplicates. It can be harder when there is very little movement but a lot of compression artifacts.

And what to do now?

Upload samples.

[johnmeyer]

poisondeathray, you may have hit the nail on the head: 23.976 --> 29.97 conversion.

Back in the late 90's a lot of camcorder were touting their "24 frame per second" ability. People thought this was a good thing, because it matched the 24 fps of movie film, so they thought they could suddenly be able to take videos that looked like Hollywood movies. I won't go into all the reasons why that assumption was wrong, but the key thing is that all the TV sets of that era were hard wired to play 30 fps (29.97 fps). So to make this 24 fps material play, most camcorders, like the Sony TRV-11 that I still own, wrapped the 24 (23.976) video into a 30 fps stream by adding duplicates. Basically, they telecined the video.

That's as far as I can go without actually seen a sample.

As everyone else has said, it is totally pointless for the OP to post any further without uploading a 5-10 second sample. Speaking for myself, I won't bother to speculate further.

The uploaded video should NOT be re-encoded: use software that does "smart rendering" or "lossless cuts," or just pick a sample that is already really short. If the problem is caused by what I just described, the result is very similar to what you get when film is telecined (dup frames/fields added for showing on broadcast TV) and the solutions involve inverse telecine (IVTC), something that is very straightforward and which will result in perfect video. If it is the capture issue I described in an earlier post, it may be tougher to deal with. And, if it is truly just duplicated frames, without any drops, then that too is pretty easy to fix using the filldrops() function developed by Mugfunky and which I updated to the later version of MVTools2 and also adapted to deal with interlaced video. However, if you use that function, the audio will go out of sync and something will have to be done about that.

[nji9nji9]

I do hope not to offend anyone if I repeat for the 5th time
that I do further algorithmic analysis on movies.
For that I do have to understand what can cause this duplicating effect
in general, to be able to identify and handle it.
I don't want to restore my one precious grandma's 80.birthday movie.
It wouldn't help me at all to have one of the concerned movies restored.

Moreover - in respect to the trust of the shown people in the movies,
I would never make them public, not 10 seconds, not 1 frame.

The movies I discovered the dupe effect in do not suffer from telecine
(as I already pointed out at my post #6, paragraph IVTC).
In all of them, if I have a close look at the (exported) frames,
in sequences of evenly movements there are only dupes, no drops
(also already said, in post #8).
None of the given explanations fits to that effect.

My problem is to identify the dupes (and distinguish from non-motion frames).

Well, I learned something.

Still don't know how to handle the problem (maybe at last have to turn
to AVISynth / filldrops(). (But there are no drops!).

Thank you.

[poisondeathray]

The movies I discovered the dupe effect in do not suffer from telecine
(as I already pointed out at my post #6, paragraph IVTC).
In all of them, if I have a close look at the (exported) frames,
in sequences of evenly movements there are only dupes, no drops
(also already said, in post #8).
None of the given explanations fits to that effect.

Can you exclude that it was edited in any way ?

Can you be sure it's not a source parsing issue or timecodes issue ?

Is the sequence natively another framerate ? Eg sometimes an edit will "slow motion" by introducing duplicates or other cadences on purpose

100's of possible explanations . 100's of possible causes. Just search this forum . You can't expect people to list every case based on your sparse description

My problem is to identify the dupes (and distinguish from non-motion frames).

There are filters that identify similarity between frames and can overlay that info (so you can go frame by frame to debug) , or print out a text file . If you just want to identify, you can leave it like that - it's all identified. But they can also decimate by % similarity based on threshold settings (e.g. drop duplicate or triplicates, or strings up to 16 of 97.65% similarity). Maybe for another video you want 95% similar duplicate frames dropped. You can change the settings

E.g. avisynth 's dedup filter and any of it's derivatives

If the dupes are meant to be there for sync, this means decimating will make you go out of sync if the framerate isn't adjusted . But dedup can produce VFR timecodes to keep everything in sync

But if the motion was meant to be smoother, then interpolating scripts can synthesis "inbetween" frames (but often with artifacts)


I'll mention some typical problem scenarios. What if camera is stationary, but intended e.g. pointing to a wall. e.g. blank screen or part of credit section. Sometimes duplicates are meant to be there too. So be careful

Good luck

[nji9nji9]

THAT'S an answer. Thank you.

Not knowing much about all that, I thought there would be a "typical", probable cause.

But 100s ...!

BTW: With "non-motion" frames I meant the stationary scenes.

[jagabo]

if I have a close look at the (exported) frames,
in sequences of evenly movements there are only dupes, no drops

Did you check for drops at the duplicate frames? For example, instead of frame numbers 0,1,2,3,4,5, you have 0,1,2,2,4,5. Frame 3 was dropped and replaced with a copy of frame 2. This often happens with video capture. And it's different than the other type of dropped frame where you may have something like 0,1,1,2,3,5. Here frame 1 was duplicated to make up for the dropped frame 4.

In any case, dups can be detected by summing/averaging abs(current[y,x]-previous[y,x]) over the entire frame. Then comparing to a threshold value. In the case of exact dups the value will be 0. Near dups (dups but with different compression artifacts or other noise) will be small. The problem arises when there are only very small changes from frame to frame. Say, a still shot of a clock with just an hour hand and a minute hand -- there's no detectable motion from frame to frame, even every frame is technically unique.

Attached is a sample created with AviSynth using such an algorithm. The average diff (Y channel only) is printed at the top left. If you single step through the video you'll see that the dup frames all have diff less than 3.0. The script line that calculated the diff and wrote it to the picture was:

ScriptClip(last, "Subtitle(String(YDifferenceFromPrevious))")

[manono]

None of the given explanations fits to that effect.

Nonsense. I gave you a plausible explanation in my first post and nothing I've read from you since has done anything to change my mind:

It's often done as a way to bring up the framerate from a lower initial framerate.

And you accused me of misunderstanding. They can only be returned to the original framerates using AviSynth filters. Perhaps perfectly, perhaps not. The spacing of the dupes you detailed in the first post is exactly the kind of thing done for lower framerate silent films to have them play properly on DVDs. I still suspect the source of these things to be home movies but one can only guess in the absence of samples. And you will never fix them using the decimators in VDub2.

Moreover - in respect to the trust of the shown people in the movies,
I would never make them public, not 10 seconds, not 1 frame.

What a load of crap. Even the people asking for help with their porn collections find places without anything serious going on. You produce samples or you get no real help, it's as simple as that. So far, in the absence of samples, people can only speculate.

[johnmeyer]

I don't want to restore my one precious grandma's 80.birthday movie.
It wouldn't help me at all to have one of the concerned movies restored.

Then what the heck is the point of this whole thread, and why am I wasting my time trying to help???

Moreover - in respect to the trust of the shown people in the movies,
I would never make them public, not 10 seconds, not 1 frame.

In that case, you cannot be helped.

Still don't know how to handle the problem (maybe at last have to turn
to AVISynth / filldrops(). (But there are no drops!).

Filldrops replaces each duplicate frame with an new frame synthesized from the adjacent frames, using motion estimation. The frame count does not change. It is all done automatically: you run it, the dup frames are gone.

However, I am not wasting one more second with this person. I totally agree with the sentiments and disgust posted by manono in the last paragraph of his recent post:

What a load of crap. Even the people asking for help with their porn collections find places without anything serious going on. You produce samples or you get no real help, it's as simple as that. So far, in the absence of samples, people can only speculate.

[nji9nji9]

Found a public access example:

https://youtu.be/L4kQdOWojeQ

What I see is a "repeat pattern" of 6 frames.
Which is "disturbed" sometimes.
No dropped frames.

My questions:
What caused it?
What can be done?

Greetings

[poisondeathray]

Found a public access example:

https://youtu.be/L4kQdOWojeQ

What I see is a "repeat pattern" of 6 frames.
Which is "disturbed" sometimes.
No dropped frames.

My questions:
What caused it?
What can be done?

Greetings

This is a duplicate inserted every 5th, 6th frame (5th and 6th are repeats) due to a framerate conversion from "25p" to "29.97p" . It might be slightly "contaminated" by youtube and lossy compression (the duplicates might have slight minor difference)

You can use avisynth to drop the duplicates and return the original frames and framerate . But it requires re-encoding. It's adaptive, so if there is a cadence break it should adapt

Tdecimate(cycler=1, cycle=6)

If there are other sections that have slightly different patterns (varying patterns indicating varying framerates, or problems like edits after the framerate conversion) , you can use dedup to make a VFR video to keep sync . But only software media player will 100% play back properly , VFR is not supported by some devices like some older TV's, DVD players . But newer devices like phones can play it back

[nji9nji9]

Thank you again.

The word "adaptive" is the point I didn't know that decimate filters could do.
How good will that adaption be ... if the "intermediate disturbances" are distinct?
And then there are the "non-motion" frames from the film content itself,
that have to be distinguished from the "artificial inserted dupes".

What may be the reason for this 6-dupes pattern?
It is not TC/ IVTC at last.
(I've also seen 2-patterns, and 7-patterns in other movies.)
Conversion?

And - last question -
(I mentioned earlier in the thread)
if possible would like to avoid the new tool "avisynth",
I'm doing with VirtualDub2 since long.
Maybe there is an adaptive decimate filter too?

Greetings!

[poisondeathray]

The word "adaptive" is the point I didn't know that decimate filters could do.
How good will that adaption be ... if the "intermediate disturbances" are distinct?
And then there are the "non-motion" frames from the film content itself,
that have to be distinguished from the "artificial inserted dupes".

If it's a wanted duplicate (e.g. camera not moving, pointing at wall, static picture), it will try to maintain the cadence . It's not 100% perfect, it can make errors

But there are many settings you can adjust and customize for a particular source, such as thresholds for duplicate detection (you can run in debug mode while tweaking the settings) , and there are overrides you can specify as well

Can you point in that sample what you are referring to by "intermediate disturbances" so that we are "on the same page" . eg. What time frame in the youtube video , and describe them? Are you referring to the compression issues like blocking on some frames ?

For very complex patterns, always changing, or edits that change the underlying framerate you're going to have to upload a real sample or figure it out yourself. There is no way around it.

What may be the reason for this 6-dupes pattern?

I already mentioned it above - 25p to 29.97p conversion

And - last question -
(I mentioned earlier in the thread)
if possible would like to avoid the new tool "avisynth",
I'm doing with VirtualDub2 since long.
Maybe there is an adaptive decimate filter too?

Not in vdub. You need avisynth if you want all the control.

[nji9nji9]

Probably it's because I don't know too much about movie issues, so ...
Of course I see that the framerate changes to that values if you delete every 6th frame on average.
But why this values?
And why - in other movies I've seen - every 2nd, or every 7th?

I tried a decimate filter for VD once
http://rationalqm.us/telecide.html
(don't know, if it's adaptive).
It produces a list of to-be-removed frames.
But - laugh at it loud - I haven't found how to "load the generated avisynth script" as described.

[manono]

I haven't found how to "load the generated avisynth script" as described.

File->Open video file...

And why - in other movies I've seen - every 2nd, or every 7th?

Videos have many many different original framerates and many many different ways to increase those framerates to something else.

[nji9nji9]

I haven't found how to "load the generated avisynth script" as described.

File->Open video file...

Yes, I KNEW it was something to be laughed at

Still, VD2 gives a dialog "file open error" then: "AVI Import filter error (Unknow) 80040154" ...
???

[manono]

With questions like this, always include your complete AviSynth script. And what kind of an AVI is it? If you don't know then post the text file from Media Info

[jagabo]

I haven't found how to "load the generated avisynth script" as described.

File->Open video file...

Yes, I KNEW it was something to be laughed at

Still, VD2 gives a dialog "file open error" then: "AVI Import filter error (Unknow) 80040154" ...
???

AviSource() is for AVI files. Your source must be an AVI file (not some other kind of file renamed .AVI) and you must have a VFW decoder installed for the codec used in the file.

If your file is not an AVI (your post originally mentioned mp4) file you need a different source filter.

[shekh]

I haven't found how to "load the generated avisynth script" as described.

File->Open video file...

Yes, I KNEW it was something to be laughed at

Still, VD2 gives a dialog "file open error" then: "AVI Import filter error (Unknow) 80040154" ...
???

This is typical error when you don't have avisynth (or it is improperly installed)

[nji9nji9]

Don't have a computer here, sorry.

OK, no avisynth ...

[jagabo]

First 6+ minutes:

LWLibavVideoSource("Der Kommissar 28 Drei Tote reisen nach Wien - HD-L4kQdOWojeQ.mkv") 
TDecimate(Cycle=6, CycleR=1)
Trim(0,9999) # first 10000 frames (after decimation)

[nji9nji9]

First of all:
Thank you all of you for your help!

I think that I do understand now what may cause (real) dupes,
and which are the means.

The example of avisynth's TDecimate is quite impressive.
The adaption of shifted/ broken cadences seems to do really good.

Actually there exists a decimate filter for VirtualDub (from Donald Graft).
(In my post above
https://forum.videohelp.com/threads/394366-Duplicates-Frames-Cause-and-means#post2562202
I made the error of pointing to his telecine filter. Sorry.
I do meant his decimate filter:
http://rationalqm.us/decimate.html
But this filter is outdated in many ways (the author mentions several points).
It seems to be kind of adaptive, but - as it is written for an early version of VirtalDub -
has to use avisynth to delete frames.
But actual VirtualDub(2) can delete frames.

I wonder why there is no filter for the "easy-to-use" VirtualDub
for the task of adaptive decimation ...
but one has to dip deep into something like avisynth scripts,
packing into AVI container, enabling VfW codec, ...
Big hurdle.

Please don't let me misunderstood:
No complaining, no accusation, ...
Just stating from a ... well ... newbies'/ user's view.

Will see what I can do on the VirtualDub2's filter front.

Thanks again