de-duplicating / decimating ProRes losslessly?

[chconnor]

Hi -- I have a bunch of footage in ProRes 422. It was Super 8 footage that a place transferred for me, and it's in 29.97, so they duplicated frames (1-2-2 pattern) to match the original 18fps frame rate (contrary to what we were hoping for). (To be clear, not telecined: all the frames are clear single film frames.)

I want to de-duplicate it to it's native 18fps. I've found this answer for using ffmpeg's mpdecimate filter.

...but I want to do this losslessly. Since ProRes is all intra-frame, shouldn't it be possible to de-duplicate it without transcoding? Any tips on how to approach that? Or is ffmpeg smart enough to do that if it's reading in ProRes and writing out ProRes?

Thanks for thoughts!
-c

P.S. here's the footage info:

$ ffprobe file.mov 
ffprobe version 3.0.7-0ubuntu0.16.10.1 Copyright (c) 2007-2017 the FFmpeg developers
  built with gcc 6.2.0 (Ubuntu 6.2.0-5ubuntu12) 20161005
...
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file.mov':                                                                                                      
  Metadata:                                                                                                                                                                                                      
    major_brand     : qt                                                                                                                                                                                         
    minor_version   : 537199360                                                                                                                                                                                  
    compatible_brands: qt                                                                                                                                                                                        
    creation_time   : 2017-03-31 13:10:28                                                                                                                                                                        
    com.apple.proapps.logNote: etc                                                                                                                                                             
    com.apple.proapps.reel: 001                                                                                                                                                                                  
    com.apple.proapps.scene: Scene 01                                                                                                                                                                            
    com.apple.proapps.shot:                                                                                                                                                                                      
    com.apple.proapps.angle:                                                                                                                                                                                     
  Duration: 00:11:16.31, start: 0.000000, bitrate: 207087 kb/s                                                                                                                                                   
    Stream #0:0(eng): Video: prores (apch / 0x68637061), yuv422p10le(bt709), 1920x1080, 207081 kb/s, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc (default)                                          
    Metadata:                                                                                                                                                                                                    
      creation_time   : 2017-03-31 13:10:28                                                                                                                                                                      
      handler_name    : Apple Alias Data Handler                                                                                                                                                                 
      encoder         : ProRes 422 HQ                                                                                                                                                                            
      timecode        : 00:00:00;00                                                                                                                                                                              
    Stream #0:1(eng): Data: none (tmcd / 0x64636D74) (default)                                                                                                                                                   
    Metadata:                                                                                                                                                                                                    
      creation_time   : 2017-03-31 13:10:28                                                                                                                                                                      
      handler_name    : Apple Alias Data Handler                                                                                                                                                                 
      reel_name       : time                                                                                                                                                                                     
      timecode        : 00:00:00;00                                                                                                                                                                              
Unsupported codec with id 0 for input stream 1

[poisondeathray]

You would have to do it manually specifying where to cut if you use a decimation filter. Because when a filter is applied (ffmpeg or avisynth) , frames are decoded (decompressed to uncompressed 10bit422 YUV for prores HQ) . You can never get back the original prores compression . Recompressing it with prores would be lossy .

It might be possible if you output a decision list and use that to specify script cuts in stream copy mode (-c:v copy)

[chconnor]

Thanks! -- that's what I was afraid of.

Are you aware of any scripting way I could make it happen? I thought I read about some plugins for NLE's (which I don't own) that allow you to open footage in deduplicated form... I'm using Davinci Resolve... wonder if there's a scripty way I could do it.

Or a script I could write for Blender? But maybe it's not capable of the no-transcoding cuts, since it probably uses ffmpeg anyway...

Or if you knew of a relatively simple code library? I'm no C hacker, but if there was e.g. a python or java library for lossless cuts in ProRes I could write a program to do it...

[poisondeathray]

You would need software that can "smart render" prores. So MAC version of PP (PC won't do it), FCP/X, etc... and if you feed it an EDL or AAF it should work ("cuts only" editing will pass through frames untouched) . But the formatting of EDL's and AAF's vary

In theory ffmpeg could do it too with -vcodec copy -acodec copy , with -t and -ss , and appending the segments with concat demuxer . It doesn't have direct edl input like mplayer .

There might be some advanced scripting you can use to get it working
eg. maybe something like this
https://hackmemory.wordpress.com/2013/04/08/lightning-fast-video-splitting-script/


Do you really need it back as prores ? It's not a huge quality hit, and there are established workflows for lossless intermediates if you really need that

[shekh]

Possible in vdubFM, it can pass prores without recompression. I don't know if there is a filter to remove exactly the frames you want. Maybe just try "convert to fps".

[chconnor]

In theory ffmpeg could do it too with -vcodec copy -acodec copy , with -t and -ss , and appending the segments with concat demuxer . It doesn't have direct edl input like mplayer .

Interesting -- chop up the original into thousands of single-frame-length prores files, then concatenate them? I'll look further in to that...Since the concat demuxer can work off a text file list, that bodes well. Hopefully specifying the frame by timestamp is accurate enough to work (it apparently goes into microseconds) and hopefully the files themselves stay true to the 122 pattern... otherwise i'm sunk. :-)

Do you really need it back as prores ? It's not a huge quality hit, and there are established workflows for lossless intermediates if you really need that

Yeah I'll survive if I can't make it happen (the transfer house may just end up running the film again.) It's family archive stuff, and I just want to preserve best I can for posterity.

@shekh -- thanks -- good to be clued in to virtualdub filtermod -- the frame rate change looks like it can select every nth frame, but not do patterns, but it's nice to know about.

[shekh]

With luck the pattern may appear automagically, I`d try to just set "18" in the "convert to" field.

[chconnor]

It occurs to me that since the file is in 30/1.001 and the super 8 was 18, there's a chance that the duplication may not strictly follow 1-2-2 and might not duplicate a frame now and then to stay in time... i guess there's only one way to find out...

[chconnor]

With luck the pattern may appear automagically, I`d try to just set "18" in the "convert to" field.

Meaning, VDFM might have been coded to do this under the hood by detecting duplicates or something? Does that maybe seem unlikely? I suppose it'd be easy enough for me to try, and would certainly save time.

[shekh]

No, it does not detect nothing, but it will throw away some frames to make new framerate. This is old feature from VirtualDub.

[chconnor]

Some progress... I made a script that uses ffmpeg to run mpdecimate and just log the results (without writing a file):

ffmpeg -i file.mov -vf mpdecimate=max=1 -loglevel debug -f null - > mpdecimate.txt 2>&1

I then parse out the time stamps of "keep" frames from that log, and use ffmpeg to generate single-frame .mov files for each "keep" frame:

cat mpdecimate.txt | grep Parsed | grep keep | sed -r 's/^.*pts_time:(.*) drop.*/\1/' > ts.txt
cat ts.txt | awk "{printf \"ffmpeg -ss %s -t $flen -i file.mov -vcodec copy -acodec copy %08d.mov\n\",\$1,f;f++}" > ffmpeg.txt
source ffmpeg.txt

(where flen is the calculated time length of a single frame, e.g. 1/(30/1.001))

That generates (and runs) a file that has many commands like this:

ffmpeg -ss 43.0096 -t 0.033366666666666 -i file.mov -vcodec copy -acodec copy 00000768.mov

I then concatenate the resulting single-frame mov files with this (I'm leaving out a lot of the script; the original .mov is not in this same directory):

/bin/ls -1 *.mov | sed -r "s/(.*)/file '\1'/" > segmentfiles.txt
ffmpeg -f concat -i segmentfiles.txt -c copy file_dedup.mov

This almost works. :-)

It seems fine for the first few seconds, and then starts to slow down: duplicate frames are generated, frames skip back in time, etc. It seems apparent that the -ss and -t combo is perhaps not precise enough to target an individual frame (or that the log file mpdecimate is generating is not accurate, or that I'm misinterpreting it somehow.) Any tips on that?

Also, the generated result file is still 29.97, and I'm not having great success switching it to 18. The framerate changing stuff seems to require transcoding. The more elaborate route of changing timescale intimidated me a bit, and I wasn't sure if it was wise for my case. Any ideas on that?

Thanks! I appreciate the time.

[chconnor]

Hey, further clues!

If I move -ss and -t after the initial -i, it targets the frames more accurately. Seems odd to me, since they're all keyframes, and I thought -ss before -i meant that it found the nearest keyframe to that point in time. Maybe a bug? Takes a little longer with -ss after the -i, but not too bad. Edit: I thought it took longer; maybe not, since it's ProRes and shouldn't really matter? Anyway...

Still some duplicated frames in the output, though... working on tweaking mpdecimate now...

[poisondeathray]

Did you try playing with the threshold values ?

Is there audio ?

There are other non ffmpeg methods that are probably more accurate for smart decimation


For Quicktime MOV it is difficult to manipulate the timecodes and keep it "legal" for compatibility . There are 1 or more timecode tracks and open source tools have difficulty editing. You can manipulate timecodes with other containers like MP4 with tools like lsmash works, mp4fpsmod, but for MOV you basically need Mac tools like FCP / FCPX

https://developer.apple.com/library/content/documentation/QuickTime/RM/MovieBasics/MTE...diaHandle.html
https://documentation.apple.com/en/finalcutpro/usermanual/index.html#chapter=51%26sect...3%26tasks=true

[chconnor]

I think I got it -- after tweaking mpdecimate, it still seemed to be a frame targeting issue... ffmpeg's -ss and -t aren't grabbing exactly the right frame all the time... so I just nudged back each frame timestamp 10% of a frame length to make sure it wasn't falling near a frame boundary, and it worked perfectly. ffmpeg must just be going to the next frame, rather than the nearest, in either -ss mode (meaning, before or after -i).

So the only remaining problem is the frame rate... thoughts?

[chconnor]

Did you try playing with the threshold values ?

Yep, thanks -- in fact I've switched over to not using hi at all, because even between duplicate frames there frequently seems to be at least one 8x8 block that crosses it. I'm currently using a lower "lo" threshold (64*3) and frac of .4. Might need tweaking per file, I don't know...

Is there audio ?

No, which is nice.

There are other non ffmpeg methods that are probably more accurate for smart decimation

Interesting -- any names/links?

For Quicktime MOV it is difficult to manipulate the timecodes and keep it "legal" for compatibility.

That's disappointing... thanks for the info. I know there is a ton going on under the hood of video codecs, but the difficulty in changing framerates really comes as a surprise. Naively one would think it would just involve tweaking a header value.

I'll post the script when it's done in case anyone else is interested.

[poisondeathray]

For decimation there are a bunch of avisynth methods . If you have no audio it's one thing less to worry about and more approaches that you could use
http://avisynth.nl/index.php/External_filters#Duplicate_Frame_Detectors

This isn't the full list, and many of the decimation filters have metrics output
http://avisynth.nl/index.php/External_filters#IVTC_.26_Decimation

They definitely work , people in similar situations as you use avisynth commonly for the decimation part . The QT/Prores "smart render" cuts is a new twist however. I'm interested if you can post the script(s)



AFAIK only FCP/FCPX / cinema tools can do framerate changes to MOV/quicktime without re-encoding reliably. Blame Apple.

You can do it easily to other formats (just change header, or mux in new timecode track) ; and MOV is just a MP4 derivative you would think the same MP4 based tools for TC manipulation would work, but they don't.

[chconnor]

Ok, I made a blog post about the bash script, which I called "lld"; link to the script is on that page. I'd appreciate any corrections to my thinking on it.

Thanks again for the help!

http://lacinato.com/cm/blog/26-lld-losslessly-decimate-de-duplicate-on-linux-prores-etc

[poisondeathray]

Thanks for the follow up post & script