Delete duplicated frames from silent film?

[Spiny Norman]

There's this silent film I have as mpg1 where every 2nd and 3rd frame are the same (though not in the first 4 frames for some reason).

I converted it to avi and then I wanted to remove the duplicated frames and reduce the frame rate. Seemed a logical step that would reduce the file size and make it easier to upload. There's no audio track so nothing to worry about there.

Vdub(mod) has a function for this, but it doesn't seem to work? After processing every 2nd and 3rd frame were still the same.

Did I overlook something? It would be a great help if it could be done because of upload limits. Preferably without full re-encoding. Thanks.

[davexnet]

I attempted this a few years back with the public domain film Nosferatu,
lowered the rate from 25 to approx 19, using avisynth/fdecimate.

It does need a full re-encode.

[jman98]

If you need a smaller file to upload then you need to use a lower bit rate to encode it with. Trying trickery like changing the frame rate won't accomplish your goal.

In case you don't know, it wasn't until sometime in the late 1920s I think that Hollywood finally standardized on 24 fps. Silent films can have all kinds of actual frame rates. I think 18 fps is the lowest I've heard of. Things like 20 and 22 fps were also known.

Do a test encode with something at say 29.97 fps and the same thing at 24 fps at the same bit rate and you'll see very little difference in the file sizes.

[Spiny Norman]

Strange, I would expect it to be significantly smaller if you delete 1/3rd of all frames. Especially without an audio track.

[smrpix]

I would expect it to be significantly smaller if you delete 1/3rd of all frames.

Depending on the original encoding, you are simply deleting a miniscule instruction that says in effect, "frame 102= frame 101."

[sanlyn]

Every frame in a video encode is not a complete image. Most Video encodes achieve higher levels of compression by storing the video as groups of pictures (GOP). This is an oversimplified description, but each GOP has a key frame or index frame (I frame) that is a complete image. The other "frames" in this group aren't complete images, but are data that describes how the video has changed since the last key frame. To further reduce file size, many telecine or duped frames don't really exist: rather, you have code that tells the decoding device to repeat frames so-and-so, meaning that the data is stored only once but is displayed twice.

The reverse occurs when video is decoded for, say, denoising and color correction. A 1GB video of 150,000 partial-image frames is decoded into 150,000 complete, full-frame images. The IGB file becomes something like 15GB, or some such increase. If you make your changes and then re-encode the video, the encoding process itself not only goes back to that I-frame GOP structure, but depending on how high or low you set your bitrate, some of that incoming data is discarded from the original. Low bitrates use less of the data, higher bitrates use more. That's why most video encoding is called "lossy", and it's why careless re-encoding is frowned upon.

Say, but I think I understand what I just wrote !

[manono]

Post a sample from the source, 10 seconds or so with steady movement. You'll need AviSynth to remove the duplicate frames. It'll have to be reencoded and, as mentioned, you won't save a whole lot of space if reencoding for the same quality because duplicates take up almost no bits.

[yonkiman]

One point no one mentioned: you might still want to remove duplicate frames to get the cadence right. The OP's images go abbcddeffghh. The video might play back smoother if it was encoded as abcdefgh at 2/3s the original frame rate. I don't believe source material should be encoded as anything a sequence of unique frames - that's making assumptions about how it will be played back, assumptions that may be incorrect. It was really only a good idea in the analog TV days.

I've got a DV file with a crazy cadence (from an old Super 8 film). Cadence almost seems random. I just looked at a clip and here are the number of identical frames in sequence: 3212222132122212312!!! It's the 3s next to the 1s that kill me - one frame gets 3x more screen time than the adjacent one, rather than just going 22.

So I want to fix the cadence, and in my case, since each frame is individually compressed in this format, I'll probably halve my 40GB source file!

If anyone has good AVIsynth tutorial/script for fixing this (it needs to identify duplicate frames and delete them, not just decimate with a particular sequence), I'd love to see it. I'm a newb and when I tried this a year or two ago I just got frustrated. Hoping I've wizened since then...

[yonkiman]

Here's a sample .avi of the film long enough to see the weird cadence and test a script for fixing it...any suggestions appreciated!

[jagabo]

The underlying frame rate is about 16 fps. You can use TDecimate() in AviSynth to reduce the clip to unique frames:

AviSource("Super 8 Test Clip.avi") 
TDecimate(mode=2, rate=16.0)

I saw only one duplicate frame in the clip with that. You can try slightly lower rates to remove that last dup, around 15.7.

[manono]

You didn't provide enough frames to be sure, but it looks to me like the average is a dupe frame for every frame or 14.985fps. So, although there's a missing frame using this script, it's easily accounted for by there not being enough frames to decimate properly,:

Tdecimate(Mode=2,Rate=14.985,MaxNDL=4,M2PA=True)

I just saw jagabo post his response. And my results were slightly different. On second thought, since Super 8 is 16fps (?), then maybe this is really 15.984 or so. And I found 2 missing frames at 14.985fps.

Edit: Even at 15.984 fps there's one missing frame, but perhaps it'll work for the whole video.

Tdecimate(Mode=2,Rate=15.984,MaxNDL=4,M2PA=True)

[jagabo]

At 15.7 fps I got no duplicate frames and didn't notice an missing frames. But a longer clip might change that a bit.

[manono]

Lemme try 15.7...

It's the same as I got at 15.984. Just as the woman catches the ball there's a frame missing. But yes, the longer the video the more the true framerate will reveal itself. There's not quite enough here.

I'm using FFVideoSource.

[jagabo]

If I add the MaxNDL=4,M2PA=True arguments to TDecimate I get the same result as you at 15.7, a missing frame as the woman captures the ball. I also get a duplicate frame at 58-59. Normally I'd expect MaxNDL=4,M2PA=True to work better but in this particular case leaving them out appears to work better.

<edit>
Setting MaxNDL=5 fixes the problems.
</edit>

[yonkiman]

jagabo, manono - you guys ROCK! A few comments/questions:

• I called this Super 8, but it was probably regular 8. I just checked and Super 8 didn't exist until a few years after this was taken. I do have some Super 8 footage as well - the reels digitized were about 50% 8 and 50% Super 8. So from what you guys are saying I would imagine the camera was set to 16FPS?
• Is there a "dumber" way to decimate the frames? It seems like the algorithm/script you guys are using is trying to fit the frames into a specific FPS, causing dropped/duplicate frames (I think that's what frustrated me when I tried it a few years ago). I know the camera was running at a (semi- ) constant rate, so I just want all the frames in sequence, then I figure I could set the frame rate afterwards. I don't really care if it ends up 1% faster or slower - I just want all the frames. Know what I mean?

Thanks again for your help. Should I upload a longer clip or is it time for me to take it from here?

-Fred

[poisondeathray]

jagabo, manono - you guys ROCK! A few comments/questions:
[LIST][*]I called this Super 8, but it was probably regular 8. I just checked and Super 8 didn't exist until a few years after this was taken. I do have some Super 8 footage as well - the reels digitized were about 50% 8 and 50% Super 8. So from what you guys are saying I would imagine the camera was set to 16FPS?

Yes, it looks like 16 FPS

[*]Is there a "dumber" way to decimate the frames? It seems like the algorithm/script you guys are using is trying to fit the frames into a specific FPS, causing dropped/duplicate frames (I think that's what frustrated me when I tried it a few years ago). I know the camera was running at a (semi- ) constant rate, so I just want all the frames in sequence, then I figure I could set the frame rate afterwards. I don't really care if it ends up 1% faster or slower - I just want all the frames. Know what I mean?

Yes, there are other "pure" decimation functions like multidecimate, fdecimate, mdec2, dedup, a few others

It might be a case of ".dll hell" (sometimes people get different results with avisynth for various reasons), but I get the 1st frame missing with Tdecimate at 16 (61 frames), 2 frames missing at 15.7 (60 frames) .

If you check with something like dedup and look at the log there should be 62 unique frames. That would work out to be about 16.1-16.2 fps on this short sample - I bet it will even out on a longer sample to be 16. But you can assume any frame rate with pure duplicate decimation - the only thing for you to do is determine the threshold value that should be used (you need to adjust it so no duplicates are retained, no "good frames" are dropped)

Since it's a "relatively clean" transfer , you don't get much contamination from compression noise or grain which can screw up the metrics (noise on duplicate frames can make it "look" like duplicate frames are different) . So if this is a representative sample , it's easy to differentiate true duplicates without getting false positives or negatives with high accuracy . (It should be 100% accurate looking at the log)

[manono]

Setting MaxNDL=5 fixes the problems.

Yeah, you're right. No idea why, though. I included M2PA=True for yonkiman and his whole video. There's really no use for it here with so few frames.

Should I upload a longer clip or is it time for me to take it from here?

You can take it from here and play around with it. I think this 8mm film is supposed to be 16fps, and since the source is 29.97fps (and not 30fps), I'd try 15.984 first and check for dupe and/or missing frames. Then maybe 16fps or the 15.7 jagabo suggested. It's always better for smoothness to have a few duplicate frames here and there, rather than missing frames. With the decimation pattern not being consistent, it may be difficult to get it exactly right.

[jagabo]

So from what you guys are saying I would imagine the camera was set to 16FPS?

Nominally, yes. But neither the camera nor projector is very accurate as far as frame rates are concerned.

Is there a "dumber" way to decimate the frames?

Yes, but the results will be much worse. ChangeFPS(15.7).

[yonkiman]

Yes, there are other "pure" decimation functions like multidecimate, fdecimate, mdec2, dedup, a few others

Which one would you recommend for this project?

Since it's a "relatively clean" transfer , you don't get much contamination from compression noise or grain which can screw up the metrics (noise on duplicate frames can make it "look" like duplicate frames are different) . So if this is a representative sample , it's easy to differentiate true duplicates without getting false positives or negatives with high accuracy . (It should be 100% accurate looking at the log

Yeah, the duplicate frames appear to be digital-exact duplicates. Which makes it all the harder for me to understand why it has this crazy-ass cadence.

It's always better for smoothness to have a few duplicate frames here and there, rather than missing frames. With the decimation pattern not being consistent, it may be difficult to get it exactly right.

I don't think in this case it's always better for smoothness - that's only true if you know it will be played back at x FPS... I'd rather have the software playing the video back do the conversion from 16 or 15.7xx to whatever the connected display wants. What gives you the best picture at 60Hz might not give you the optimal picture at 120Hz (where you could spread the error out more evenly).

But I could be completely wrong.

Thanks again.

[poisondeathray]

Yes, there are other "pure" decimation functions like multidecimate, fdecimate, mdec2, dedup, a few others

Which one would you recommend for this project?

Since it's a "relatively clean" transfer , you don't get much contamination from compression noise or grain which can screw up the metrics (noise on duplicate frames can make it "look" like duplicate frames are different) . So if this is a representative sample , it's easy to differentiate true duplicates without getting false positives or negatives with high accuracy . (It should be 100% accurate looking at the log

Yeah, the duplicate frames appear to be digital-exact duplicates. Which makes it all the harder for me to understand why it has this crazy-ass cadence.

Not sure why it has that cadence . Maybe provide some details on the projector and capture setup

With no audio and sync to worry about, I would use dedup in non VFR mode (ie. just decimate the duplicates, no use of timecodes, then assume any frame rate you want, but the "proper" frame rate is going to be around 16).

It requires 2 passes, but looking at the metrics (if this is a representative sample), it's going to be very accurate with no duplicates retained, no "good frames" dropped. Like you said, the duplicates are virtually identical (no grain or noise or other issues).

It's not a matter of fitting to a specified frame rate - it's measuring the difference between frames or strings of frames (you can set limits), and if it's above a certain value, the frame is allowed to pass, if it's below a certain value the frame gets dropped. When used without the timecodes function, it's "framerate agnostic", just pure decimation of duplicates

For perspective, with "noisy" sources, or ones that have shutter issues and "blends", you might have a very narrow window , maybe between 0.1-2 for the threshold before other pre-treatments for metrics like denoising. You run a fine line between dropping "good " frames and keeping duplicates. Because your duplicates are "clean" you have about 0-14 window (that's a huge margin of error - in my experience that essentially means no error)



Other things I would take care of:
1) If you're using a libav based decoder for mjpeg, it will decode mjpeg as full range YUV (Your video is using YUV 422) . So I would "legalize" the values. If you're using a decoder that clamps values, the levels adjustment isn't necessary but you probably still have to adjust levels slightly
2) Treat the combing with something like Vinverse() or other filters

[yonkiman]

Haven't dropped this, just haven't been able to get AVIsynth to process the file correctly. Most other video filetypes work, but this (most likely due to PDR's point #1, directly above) just plays back with a black screen (audio is there for clips that have it). Thought I was using libav (it's the one selected in the ffmpeg settings control panel). So I'm not having a level problem (yet) - I'm having a "everything is black" problem.

Fortunately wont have to use Vinverse() since the film was scanned in progressive HD...

[jagabo]

haven't been able to get AVIsynth to process the file correctly. Most other video filetypes work, but this (most likely due to PDR's point #1, directly above) just plays back with a black screen

What source filter are you using? AviSource() should work if you use ffdshow and enable its VFW MJPEG decoder. You can then use ColorYUV(levels="PC->TV") to convert from full 0-255 to 16-235. Or manually adjust the black/white levels.

[jagabo]

I would use dedup in non VFR mode

I didn't realize it had a non VFR mode. Thanks for pointing it out. It may be useful for some other projects I'm working on.

[yonkiman]

What source filter are you using? AviSource() should work if you use ffdshow and enable its VFW MJPEG decoder.

Did that. Just reinstalled latest version of ffdshow and made sure libavcodec was selected for MJPEG for both VFW and "video decoder".

Maybe I'm doing something wrong. Here's my script (only using now fadeout because I'm following the getting started tutorial):

AVISource("F:\AVI Editing\Super 8 Test Clip.avi")
FadeOut(10)

Then I execute the script (actually double-click on it), launching AviSynth. For other movies with different encoding, my player then launches and I can see the video and it fades the last 10 frames. But for the file I posted, my player still launches, but I just get it's default background, as if it is playing an audio file. I don't get an error - it loads the file OK but then doesn't seem to send any video to the player.

What am I doing wrong?

You can then use ColorYUV(levels="PC->TV") to convert from full 0-255 to 16-235. Or manually adjust the black/white levels.

Thanks - I can't wait until that's my problem!

[poisondeathray]

Open the .avs script in vdub or avspmod

Fortunately wont have to use Vinverse() since the film was scanned in progressive HD...

You would think that would be the case, but there is combing that can be treated. Something is messed up with this scan or somewhere in the workflow - it should not look like that. You should provide more information - is this the "native" file or was it processed in some way ?

Even look at mediainfo, the mjpeg is using interlaced encoding (Even though the actual content is progressive)
Scan type : Interlaced
Scan order : Top Field First

(You'll see the ombing when you actually get avisynth working)



Alternatively you can use FFMS2 (ffmpegsource2) to open up the video . For the most part , it doesn't rely on external codecs or splitters so it should work more consistently across different configurations and systems
https://code.google.com/p/ffmpegsource/

e.g.

FFVideoSource("Super 8 Test Clip.avi")

[yonkiman]

Open the .avs script in vdub or avspmod

Will try that next.

Fortunately wont have to use Vinverse() since the film was scanned in progressive HD...

You would think that would be the case, but there is combing that can be treated. Something is messed up with this scan or somewhere in the workflow - it should not look like that. You should provide more information - is this the "native" file or was it processed in some way ?

Well crap. I paid medium-big bucks a few years ago to have all this Reg8 and Super8 scanned to HD. This is the native file I received. Well wait, this is the native file split into 2 avi files by vdub in "Direct Stream Copy" mode. That shouldn't mess with the encoding, right? I don't have any more info other than what's embedded in the file. Looking frame-by-frame in vdub...it doesn't LOOK interlaced...

Even look at mediainfo, the mjpeg is using interlaced encoding (Even though the actual content is progressive)
Scan type : Interlaced
Scan order : Top Field First

(You'll see the ombing when you actually get avisynth working)

Wonderful.

Alternatively you can use FFMS2 (ffmpegsource2) to open up the video . For the most part , it doesn't rely on external codecs or splitters so it should work more consistently across different configurations and systems
https://code.google.com/p/ffmpegsource/

e.g.

FFVideoSource("Super 8 Test Clip.avi")

OK, I'll take a look at FFMS2.

I thought this was going to be easy. I have a series of non-interlaced JPGs in MJPEG format. Some are duplicates. I want to create a new series of JPEGs with no dupes and no decoding/re-encoding of the JPG frames...

All I want to do is what I could do manually in vdub - delete every duplicate frame, without any decompression/recompression. Unfortunately I've got about 2 hours of video, so manual isn't an option.

Maybe it's best done by literally saving each frame as a jpg, deleting any duplicate jpegs, then reassembling them and putting them in the right metadata? I think I could write a python script to do that but it's a lot of work if there's another way...

I really appreciate all the help - this would take me forever to figure out on my own...

-Fred

[yonkiman]

Open the .avs script in vdub or avspmod

VirtualDub worked! Progress! Thanks!

[poisondeathray]

If you're sure you set vdub to video=>direct stream copy, then it should be same as the original

You can double check with mediainfo (view=>text) on the original file as well

To be clear: it's just encoded interlaced, but the content is progressive (there aren't separate fields) . But there is what I would call residual "combing" . Open the original video in vdub directly and you should be able to see this

As for deleting frames - I have high confidence that dedup will work with 100% accuracy on your video. The "negatives" of doing it that way are avisynth requires re-encoding because it works in the uncompressed domain. This means generation loss. But if you're going to be "fixing" things like the combing and levels (perhaps color work as well), it's a moot point

And there is no way to losslessly decompress MJPEG into JPEG photos . They are not the quite same thing, and JPEG uses lossy compression (there are "lossless" jpeg variants LJPEG, but it's probably not what you intend to use)

But you might be able to use the log file from dedup (or other duplicate detectors) to plug into another program if you wanted to keep this lossless, and don't mind the combing

[yonkiman]

To be clear: it's just encoded interlaced, but the content is progressive (there aren't separate fields) . But there is what I would call residual "combing" . Open the original video in vdub directly and you should be able to see this

Nope. I see motion blur but nothing that resembles combing (to my eye anyway). Combing = odd and even fields offset due to movement, so every other line is a bit offset from each other, right? Want to make sure we're talking about the same thing.

As for deleting frames - I have high confidence that dedup will work with 100% accuracy on your video. The "negatives" of doing it that way are avisynth requires re-encoding because it works in the uncompressed domain. This means generation loss. But if you're going to be "fixing" things like the combing and levels (perhaps color work as well), it's a moot point

And there is no way to losslessly decompress MJPEG into JPEG photos . They are not the quite same thing, and JPEG uses lossy compression

But you might be able to use the log file from dedup (or other duplicate detectors) to plug into another program if you wanted to keep this lossless

OK, thanks. I really did want to fix the "Master" video's frame issues without having to add a generation, but you're right - I can keep this as the master and fix the duplicates whenever I edit the video. Something of a pain, but at least you've shown me it can be done.

I think I'll investigate that last option: Process the dedup log file to create a script vdub can read to delete duplicate frames.

Thanks again.

[poisondeathray]

Do you see fine horizontal lines? The fields don't quite align, so you get residual combing. It's not real "combing" in the sense of interlaced fields, where each field is a different moment in time - that's real combing. That's why it's typically called "residual combing". That's why something gentle like vinverse is used to treat it instead of deinterlacing it (it's not interlaced)

I don't know how to get vdub to process the dedup log (or any log), or how to program that part (without using avisynth to remove the duplicates)