VideoHelp Forum
+ Reply to Thread
Page 1 of 2
1 2 LastLast
Results 1 to 30 of 37
Thread
  1. There's this silent film I have as mpg1 where every 2nd and 3rd frame are the same (though not in the first 4 frames for some reason).

    I converted it to avi and then I wanted to remove the duplicated frames and reduce the frame rate. Seemed a logical step that would reduce the file size and make it easier to upload. There's no audio track so nothing to worry about there.

    Vdub(mod) has a function for this, but it doesn't seem to work? After processing every 2nd and 3rd frame were still the same.

    Did I overlook something? It would be a great help if it could be done because of upload limits. Preferably without full re-encoding. Thanks.
    This is nøt å signåture.™
    Quote Quote  
  2. Member
    Join Date: Mar 2008
    Location: United States
    Search Comp PM
    I attempted this a few years back with the public domain film Nosferatu,
    lowered the rate from 25 to approx 19, using avisynth/fdecimate.

    It does need a full re-encode.
    Quote Quote  
  3. Member
    Join Date: Oct 2004
    Location: Freedonia
    Search Comp PM
    If you need a smaller file to upload then you need to use a lower bit rate to encode it with. Trying trickery like changing the frame rate won't accomplish your goal.

    In case you don't know, it wasn't until sometime in the late 1920s I think that Hollywood finally standardized on 24 fps. Silent films can have all kinds of actual frame rates. I think 18 fps is the lowest I've heard of. Things like 20 and 22 fps were also known.

    Do a test encode with something at say 29.97 fps and the same thing at 24 fps at the same bit rate and you'll see very little difference in the file sizes.
    Quote Quote  
  4. Strange, I would expect it to be significantly smaller if you delete 1/3rd of all frames. Especially without an audio track.
    This is nøt å signåture.™
    Quote Quote  
  5. Member
    Join Date: Jun 2012
    Location: USA
    Search Comp PM
    Originally Posted by Spiny Norman View Post
    I would expect it to be significantly smaller if you delete 1/3rd of all frames.
    Depending on the original encoding, you are simply deleting a miniscule instruction that says in effect, "frame 102= frame 101."
    Quote Quote  
  6. Banned
    Join Date: Oct 2004
    Location: New York, US
    Search Comp PM
    -30-
    Last edited by sanlyn; 19th Mar 2014 at 02:48.
    Quote Quote  
  7. Post a sample from the source, 10 seconds or so with steady movement. You'll need AviSynth to remove the duplicate frames. It'll have to be reencoded and, as mentioned, you won't save a whole lot of space if reencoding for the same quality because duplicates take up almost no bits.
    Quote Quote  
  8. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    One point no one mentioned: you might still want to remove duplicate frames to get the cadence right. The OP's images go abbcddeffghh. The video might play back smoother if it was encoded as abcdefgh at 2/3s the original frame rate. I don't believe source material should be encoded as anything a sequence of unique frames - that's making assumptions about how it will be played back, assumptions that may be incorrect. It was really only a good idea in the analog TV days.

    I've got a DV file with a crazy cadence (from an old Super 8 film). Cadence almost seems random. I just looked at a clip and here are the number of identical frames in sequence: 3212222132122212312!!! It's the 3s next to the 1s that kill me - one frame gets 3x more screen time than the adjacent one, rather than just going 22.

    So I want to fix the cadence, and in my case, since each frame is individually compressed in this format, I'll probably halve my 40GB source file!

    If anyone has good AVIsynth tutorial/script for fixing this (it needs to identify duplicate frames and delete them, not just decimate with a particular sequence), I'd love to see it. I'm a newb and when I tried this a year or two ago I just got frustrated. Hoping I've wizened since then...
    Quote Quote  
  9. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    Here's a sample .avi of the film long enough to see the weird cadence and test a script for fixing it...any suggestions appreciated!
    Attached Files
    Quote Quote  
  10. The underlying frame rate is about 16 fps. You can use TDecimate() in AviSynth to reduce the clip to unique frames:

    Code:
    AviSource("Super 8 Test Clip.avi") 
    TDecimate(mode=2, rate=16.0)
    I saw only one duplicate frame in the clip with that. You can try slightly lower rates to remove that last dup, around 15.7.
    Quote Quote  
  11. You didn't provide enough frames to be sure, but it looks to me like the average is a dupe frame for every frame or 14.985fps. So, although there's a missing frame using this script, it's easily accounted for by there not being enough frames to decimate properly,:

    Tdecimate(Mode=2,Rate=14.985,MaxNDL=4,M2PA=True)

    I just saw jagabo post his response. And my results were slightly different. On second thought, since Super 8 is 16fps (?), then maybe this is really 15.984 or so. And I found 2 missing frames at 14.985fps.

    Edit: Even at 15.984 fps there's one missing frame, but perhaps it'll work for the whole video.

    Tdecimate(Mode=2,Rate=15.984,MaxNDL=4,M2PA=True)
    Last edited by manono; 3rd Mar 2014 at 20:45.
    Quote Quote  
  12. At 15.7 fps I got no duplicate frames and didn't notice an missing frames. But a longer clip might change that a bit.
    Quote Quote  
  13. Lemme try 15.7...

    It's the same as I got at 15.984. Just as the woman catches the ball there's a frame missing. But yes, the longer the video the more the true framerate will reveal itself. There's not quite enough here.

    I'm using FFVideoSource.
    Quote Quote  
  14. If I add the MaxNDL=4,M2PA=True arguments to TDecimate I get the same result as you at 15.7, a missing frame as the woman captures the ball. I also get a duplicate frame at 58-59. Normally I'd expect MaxNDL=4,M2PA=True to work better but in this particular case leaving them out appears to work better.

    <edit>
    Setting MaxNDL=5 fixes the problems.
    </edit>
    Quote Quote  
  15. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    jagabo, manono - you guys ROCK! A few comments/questions:
    • I called this Super 8, but it was probably regular 8. I just checked and Super 8 didn't exist until a few years after this was taken. I do have some Super 8 footage as well - the reels digitized were about 50% 8 and 50% Super 8. So from what you guys are saying I would imagine the camera was set to 16FPS?
    • Is there a "dumber" way to decimate the frames? It seems like the algorithm/script you guys are using is trying to fit the frames into a specific FPS, causing dropped/duplicate frames (I think that's what frustrated me when I tried it a few years ago). I know the camera was running at a (semi- ) constant rate, so I just want all the frames in sequence, then I figure I could set the frame rate afterwards. I don't really care if it ends up 1% faster or slower - I just want all the frames. Know what I mean?

    Thanks again for your help. Should I upload a longer clip or is it time for me to take it from here?

    -Fred
    Quote Quote  
  16. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    Originally Posted by yonkiman View Post
    jagabo, manono - you guys ROCK! A few comments/questions:
    [LIST][*]I called this Super 8, but it was probably regular 8. I just checked and Super 8 didn't exist until a few years after this was taken. I do have some Super 8 footage as well - the reels digitized were about 50% 8 and 50% Super 8. So from what you guys are saying I would imagine the camera was set to 16FPS?
    Yes, it looks like 16 FPS

    [*]Is there a "dumber" way to decimate the frames? It seems like the algorithm/script you guys are using is trying to fit the frames into a specific FPS, causing dropped/duplicate frames (I think that's what frustrated me when I tried it a few years ago). I know the camera was running at a (semi- ) constant rate, so I just want all the frames in sequence, then I figure I could set the frame rate afterwards. I don't really care if it ends up 1% faster or slower - I just want all the frames. Know what I mean?
    Yes, there are other "pure" decimation functions like multidecimate, fdecimate, mdec2, dedup, a few others

    It might be a case of ".dll hell" (sometimes people get different results with avisynth for various reasons), but I get the 1st frame missing with Tdecimate at 16 (61 frames), 2 frames missing at 15.7 (60 frames) .

    If you check with something like dedup and look at the log there should be 62 unique frames. That would work out to be about 16.1-16.2 fps on this short sample - I bet it will even out on a longer sample to be 16. But you can assume any frame rate with pure duplicate decimation - the only thing for you to do is determine the threshold value that should be used (you need to adjust it so no duplicates are retained, no "good frames" are dropped)

    Since it's a "relatively clean" transfer , you don't get much contamination from compression noise or grain which can screw up the metrics (noise on duplicate frames can make it "look" like duplicate frames are different) . So if this is a representative sample , it's easy to differentiate true duplicates without getting false positives or negatives with high accuracy . (It should be 100% accurate looking at the log)
    Quote Quote  
  17. Originally Posted by jagabo View Post
    Setting MaxNDL=5 fixes the problems.
    Yeah, you're right. No idea why, though. I included M2PA=True for yonkiman and his whole video. There's really no use for it here with so few frames.

    Originally Posted by yonkiman View Post
    Should I upload a longer clip or is it time for me to take it from here?
    You can take it from here and play around with it. I think this 8mm film is supposed to be 16fps, and since the source is 29.97fps (and not 30fps), I'd try 15.984 first and check for dupe and/or missing frames. Then maybe 16fps or the 15.7 jagabo suggested. It's always better for smoothness to have a few duplicate frames here and there, rather than missing frames. With the decimation pattern not being consistent, it may be difficult to get it exactly right.
    Quote Quote  
  18. Originally Posted by yonkiman View Post
    So from what you guys are saying I would imagine the camera was set to 16FPS?
    Nominally, yes. But neither the camera nor projector is very accurate as far as frame rates are concerned.

    Originally Posted by yonkiman View Post
    Is there a "dumber" way to decimate the frames?
    Yes, but the results will be much worse. ChangeFPS(15.7).
    Quote Quote  
  19. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    Originally Posted by poisondeathray View Post
    Yes, there are other "pure" decimation functions like multidecimate, fdecimate, mdec2, dedup, a few others
    Which one would you recommend for this project?

    Since it's a "relatively clean" transfer , you don't get much contamination from compression noise or grain which can screw up the metrics (noise on duplicate frames can make it "look" like duplicate frames are different) . So if this is a representative sample , it's easy to differentiate true duplicates without getting false positives or negatives with high accuracy . (It should be 100% accurate looking at the log
    Yeah, the duplicate frames appear to be digital-exact duplicates. Which makes it all the harder for me to understand why it has this crazy-ass cadence.

    Originally Posted by manono View Post
    It's always better for smoothness to have a few duplicate frames here and there, rather than missing frames. With the decimation pattern not being consistent, it may be difficult to get it exactly right.
    I don't think in this case it's always better for smoothness - that's only true if you know it will be played back at x FPS... I'd rather have the software playing the video back do the conversion from 16 or 15.7xx to whatever the connected display wants. What gives you the best picture at 60Hz might not give you the optimal picture at 120Hz (where you could spread the error out more evenly).

    But I could be completely wrong.

    Thanks again.
    Quote Quote  
  20. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    Originally Posted by yonkiman View Post
    Originally Posted by poisondeathray View Post
    Yes, there are other "pure" decimation functions like multidecimate, fdecimate, mdec2, dedup, a few others
    Which one would you recommend for this project?
    Since it's a "relatively clean" transfer , you don't get much contamination from compression noise or grain which can screw up the metrics (noise on duplicate frames can make it "look" like duplicate frames are different) . So if this is a representative sample , it's easy to differentiate true duplicates without getting false positives or negatives with high accuracy . (It should be 100% accurate looking at the log
    Yeah, the duplicate frames appear to be digital-exact duplicates. Which makes it all the harder for me to understand why it has this crazy-ass cadence.


    Not sure why it has that cadence . Maybe provide some details on the projector and capture setup

    With no audio and sync to worry about, I would use dedup in non VFR mode (ie. just decimate the duplicates, no use of timecodes, then assume any frame rate you want, but the "proper" frame rate is going to be around 16).

    It requires 2 passes, but looking at the metrics (if this is a representative sample), it's going to be very accurate with no duplicates retained, no "good frames" dropped. Like you said, the duplicates are virtually identical (no grain or noise or other issues).

    It's not a matter of fitting to a specified frame rate - it's measuring the difference between frames or strings of frames (you can set limits), and if it's above a certain value, the frame is allowed to pass, if it's below a certain value the frame gets dropped. When used without the timecodes function, it's "framerate agnostic", just pure decimation of duplicates

    For perspective, with "noisy" sources, or ones that have shutter issues and "blends", you might have a very narrow window , maybe between 0.1-2 for the threshold before other pre-treatments for metrics like denoising. You run a fine line between dropping "good " frames and keeping duplicates. Because your duplicates are "clean" you have about 0-14 window (that's a huge margin of error - in my experience that essentially means no error)



    Other things I would take care of:
    1) If you're using a libav based decoder for mjpeg, it will decode mjpeg as full range YUV (Your video is using YUV 422) . So I would "legalize" the values. If you're using a decoder that clamps values, the levels adjustment isn't necessary but you probably still have to adjust levels slightly
    2) Treat the combing with something like Vinverse() or other filters
    Last edited by poisondeathray; 3rd Mar 2014 at 22:30.
    Quote Quote  
  21. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    Haven't dropped this, just haven't been able to get AVIsynth to process the file correctly. Most other video filetypes work, but this (most likely due to PDR's point #1, directly above) just plays back with a black screen (audio is there for clips that have it). Thought I was using libav (it's the one selected in the ffmpeg settings control panel). So I'm not having a level problem (yet) - I'm having a "everything is black" problem.

    Fortunately wont have to use Vinverse() since the film was scanned in progressive HD...
    Quote Quote  
  22. Originally Posted by yonkiman View Post
    haven't been able to get AVIsynth to process the file correctly. Most other video filetypes work, but this (most likely due to PDR's point #1, directly above) just plays back with a black screen
    What source filter are you using? AviSource() should work if you use ffdshow and enable its VFW MJPEG decoder. You can then use ColorYUV(levels="PC->TV") to convert from full 0-255 to 16-235. Or manually adjust the black/white levels.
    Quote Quote  
  23. Originally Posted by poisondeathray View Post
    I would use dedup in non VFR mode
    I didn't realize it had a non VFR mode. Thanks for pointing it out. It may be useful for some other projects I'm working on.
    Quote Quote  
  24. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    Originally Posted by jagabo View Post
    What source filter are you using? AviSource() should work if you use ffdshow and enable its VFW MJPEG decoder.
    Did that. Just reinstalled latest version of ffdshow and made sure libavcodec was selected for MJPEG for both VFW and "video decoder".

    Maybe I'm doing something wrong. Here's my script (only using now fadeout because I'm following the getting started tutorial):

    Code:
    AVISource("F:\AVI Editing\Super 8 Test Clip.avi")
    FadeOut(10)
    Then I execute the script (actually double-click on it), launching AviSynth. For other movies with different encoding, my player then launches and I can see the video and it fades the last 10 frames. But for the file I posted, my player still launches, but I just get it's default background, as if it is playing an audio file. I don't get an error - it loads the file OK but then doesn't seem to send any video to the player.

    What am I doing wrong?

    You can then use ColorYUV(levels="PC->TV") to convert from full 0-255 to 16-235. Or manually adjust the black/white levels.
    Thanks - I can't wait until that's my problem!
    Quote Quote  
  25. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    Open the .avs script in vdub or avspmod

    Fortunately wont have to use Vinverse() since the film was scanned in progressive HD...
    You would think that would be the case, but there is combing that can be treated. Something is messed up with this scan or somewhere in the workflow - it should not look like that. You should provide more information - is this the "native" file or was it processed in some way ?

    Even look at mediainfo, the mjpeg is using interlaced encoding (Even though the actual content is progressive)
    Scan type : Interlaced
    Scan order : Top Field First

    (You'll see the ombing when you actually get avisynth working)



    Alternatively you can use FFMS2 (ffmpegsource2) to open up the video . For the most part , it doesn't rely on external codecs or splitters so it should work more consistently across different configurations and systems
    https://code.google.com/p/ffmpegsource/

    e.g.

    FFVideoSource("Super 8 Test Clip.avi")
    Last edited by poisondeathray; 5th Mar 2014 at 15:33.
    Quote Quote  
  26. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    Originally Posted by poisondeathray View Post
    Open the .avs script in vdub or avspmod
    Will try that next.

    Fortunately wont have to use Vinverse() since the film was scanned in progressive HD...
    You would think that would be the case, but there is combing that can be treated. Something is messed up with this scan or somewhere in the workflow - it should not look like that. You should provide more information - is this the "native" file or was it processed in some way ?
    Well crap. I paid medium-big bucks a few years ago to have all this Reg8 and Super8 scanned to HD. This is the native file I received. Well wait, this is the native file split into 2 avi files by vdub in "Direct Stream Copy" mode. That shouldn't mess with the encoding, right? I don't have any more info other than what's embedded in the file. Looking frame-by-frame in vdub...it doesn't LOOK interlaced...

    Even look at mediainfo, the mjpeg is using interlaced encoding (Even though the actual content is progressive)
    Scan type : Interlaced
    Scan order : Top Field First

    (You'll see the ombing when you actually get avisynth working)
    Wonderful.

    Alternatively you can use FFMS2 (ffmpegsource2) to open up the video . For the most part , it doesn't rely on external codecs or splitters so it should work more consistently across different configurations and systems
    https://code.google.com/p/ffmpegsource/

    e.g.

    FFVideoSource("Super 8 Test Clip.avi")
    OK, I'll take a look at FFMS2.

    I thought this was going to be easy. I have a series of non-interlaced JPGs in MJPEG format. Some are duplicates. I want to create a new series of JPEGs with no dupes and no decoding/re-encoding of the JPG frames...

    All I want to do is what I could do manually in vdub - delete every duplicate frame, without any decompression/recompression. Unfortunately I've got about 2 hours of video, so manual isn't an option.

    Maybe it's best done by literally saving each frame as a jpg, deleting any duplicate jpegs, then reassembling them and putting them in the right metadata? I think I could write a python script to do that but it's a lot of work if there's another way...

    I really appreciate all the help - this would take me forever to figure out on my own...

    -Fred
    Quote Quote  
  27. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    Originally Posted by poisondeathray View Post
    Open the .avs script in vdub or avspmod
    VirtualDub worked! Progress! Thanks!
    Quote Quote  
  28. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    If you're sure you set vdub to video=>direct stream copy, then it should be same as the original

    You can double check with mediainfo (view=>text) on the original file as well

    To be clear: it's just encoded interlaced, but the content is progressive (there aren't separate fields) . But there is what I would call residual "combing" . Open the original video in vdub directly and you should be able to see this

    As for deleting frames - I have high confidence that dedup will work with 100% accuracy on your video. The "negatives" of doing it that way are avisynth requires re-encoding because it works in the uncompressed domain. This means generation loss. But if you're going to be "fixing" things like the combing and levels (perhaps color work as well), it's a moot point

    And there is no way to losslessly decompress MJPEG into JPEG photos . They are not the quite same thing, and JPEG uses lossy compression (there are "lossless" jpeg variants LJPEG, but it's probably not what you intend to use)

    But you might be able to use the log file from dedup (or other duplicate detectors) to plug into another program if you wanted to keep this lossless, and don't mind the combing
    Quote Quote  
  29. Member
    Join Date: Mar 2014
    Location: United States
    Search PM
    Originally Posted by poisondeathray View Post
    To be clear: it's just encoded interlaced, but the content is progressive (there aren't separate fields) . But there is what I would call residual "combing" . Open the original video in vdub directly and you should be able to see this
    Nope. I see motion blur but nothing that resembles combing (to my eye anyway). Combing = odd and even fields offset due to movement, so every other line is a bit offset from each other, right? Want to make sure we're talking about the same thing.

    As for deleting frames - I have high confidence that dedup will work with 100% accuracy on your video. The "negatives" of doing it that way are avisynth requires re-encoding because it works in the uncompressed domain. This means generation loss. But if you're going to be "fixing" things like the combing and levels (perhaps color work as well), it's a moot point

    And there is no way to losslessly decompress MJPEG into JPEG photos . They are not the quite same thing, and JPEG uses lossy compression

    But you might be able to use the log file from dedup (or other duplicate detectors) to plug into another program if you wanted to keep this lossless
    OK, thanks. I really did want to fix the "Master" video's frame issues without having to add a generation, but you're right - I can keep this as the master and fix the duplicates whenever I edit the video. Something of a pain, but at least you've shown me it can be done.

    I think I'll investigate that last option: Process the dedup log file to create a script vdub can read to delete duplicate frames.

    Thanks again.
    Quote Quote  
  30. Member
    Join Date: Sep 2007
    Location: Canada
    Search Comp PM
    Do you see fine horizontal lines? The fields don't quite align, so you get residual combing. It's not real "combing" in the sense of interlaced fields, where each field is a different moment in time - that's real combing. That's why it's typically called "residual combing". That's why something gentle like vinverse is used to treat it instead of deinterlacing it (it's not interlaced)

    I don't know how to get vdub to process the dedup log (or any log), or how to program that part (without using avisynth to remove the duplicates)
    Quote Quote  



Similar Threads