29.97FPS video with many duplicated frames, convert for YouTube upload

[abolibibelot]

I have a video from which I cut a 14min segment, which I subtitled in order to share it on YouTube.

The framerate is 29.97 FPS, but I can see that there are many duplicated frames (with a totally irregular pattern), and the playback is quite jerky, although there's not a lot of motion (it's a speech by George Carlin recorded by C-Span in 1999). After reading the manual for the FDecimate Avisynth plugin, I counted the number of unique frames over a 10 seconds segment : 139 if I made no mistake, which would mean that the actual framerate is about 14 FPS. First, is it possible that the original footage was recorded at such a low and non-standard framerate ?

In summer 2011 I had recorded that same footage from C-Span's website, which at the time was available as a FLV stream in 704x480 resolution (with a display resolution of 640x480), but despite several attempts with both StreamTransport and ReplayMediaCatcher I couldn't get the whole stream flawlessly, each file had missing chunks at different spots – yesterday I created a thread on that specific issue. The thing is, despite the fact that the video currently available (successfully downloaded with youtube-dl) is in a higher resolution, 768x576, than those files downloaded years ago, I can see that the quality is inferior (significant loss of detail).

Also, the framerate of those older files is 22.475 FPS[*], and they don't seem to have duplicated frames, therefore the playback is much smoother. So whoever converted the footage to generate the currently available streams apparently screwed up the framerate big time.

[*] Actually MediaInfo's output is quite confusing :

Images par seconde                       : 22,862 Im/s
Images/s mini                            : 22,222 Im/s
Images/s maxi                            : 100,000 Im/s
Images/s d'origine                       : 22,475 Im/s

VirtualDub2 reports the framerate as “66.67 FPS” for one of these files but “22.50 FPS” for another.
VLC Media Player reports either “22.475” or “22.500”.

So my options are either :
– using the older files, which all (I kept 5 of them) have missing chunks but may be fixable if I manage to combine them and complete the holes to hopefully regenerate the complete video, with a smooth playback and a decent sharpness ;
– using the recently downloaded file from the currently available stream in 768x576, complete, but with a softer picture, and, more annoyingly, with a jerky playback.

If I go with option 2, is it wise to try to make the playback smoother by decimating the duplicated frames, or will the video be converted again to a standard framerate by YouTube anyway, making any such attempt totally moot ?

By any chance, could there be a better source for this recording available somewhere ?

Excerpt from one of the older downloads (704x480, 22.475 FPS) :
[Brain Droppings] - C-SPAN Video Library 11m19s920-25m30s285.mp4
Excerpt from the newer download (768x576, 29.97 FPS) :
GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].mp4
(This segment's content is highly worthy in its own right ! And could apply in large part to many other countries...)

[jagabo]

This gave smooth video with audio ~in-sync from the second mp4:

a = LSmashAudioSource("D:\Downloads\GCarlin.mp4") 
v = LSmashVideoSource("D:\Downloads\GCarlin.mp4").AssumeFPS(29.97)
AudioDub(v,a)
DelayAudio(0.200)

Small thumbnail version attached.

[abolibibelot]

What does it change, since the framerate is already 29.97 FPS ?
And isn't it more accurate to set the framerate as a fraction like AssumeFPS(30000,1001) ?
Why does it introduce a 200ms delay ?

EDIT : Also, an unrelated issue, when I start an encode with ffmpeg [32b, a few months old] with an AVS script as source I get a warning saying that it's impossible to start the program because d3dx9_43.dll is missing, yet as soon as I close the warning window it proceeds anyway and the outcome of the encode seems correct. Is this a known issue ?

[jagabo]

What does it change, since the framerate is already 29.97 FPS ?

LSmashVideoSource thinks the frame rate is 1.733 fps. So AssumeFPS just makes AviSynth assume the frame rate is 29.97 fps. Even though LSmashVideoSource gets the frame rate wrong it gets every frame without any dups or drops. VFR video is full of problems like this. It was a pretty easy fix in this case.

And isn't it more accurate to set the framerate as a fraction like AssumeFPS(30000,1001) ?

Sure. My point was that you could open the video with no dups or drops. And the frame rate was about 29.97. You can fine tune if if you want.

Why does it introduce a 200ms delay ??

The original video has the audio off by about 200 ms. I adjusted it. It may still be a little off. You can work on getting it more precise if you want.

EDIT : Also, an unrelated issue, when I start an encode with ffmpeg [32b, a few months old] with an AVS script as source I get a warning saying that it's impossible to start the program because d3dx9_43.dll is missing, yet as soon as I close the warning window it proceeds anyway and the outcome of the encode seems correct. Is this a known issue ?

There is probably a filter in your plugins folder that requires that dll and it failed to load. But since you aren't actually using that filter you can continue with your encoding. Try using Avisyth Info Tool to diagnose the problem.

[abolibibelot]

I don't get it. I adapted my script based on your advice :

V = LWLibavVideoSource("K:\GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].mp4").BicubicResize(640,480).AssumeFPS(29.97)
A = LWLibavAudioSource("K:\GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].mp4")
AudioDub(V, A).FadeIn(20).FadeOut(20).DelayAudio(0.200)
TextSub("K:\GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].srt")

The playback is just as jerky in VDub2 and the resulting encoded file is strictly identical to the previous one made without AssumeFPS (same checksums – the audio delay did nothing because I copied the audio from the source file for the encode, considering that it doesn't require the fade-in / fade-out so it avoids a generation loss).
If I load the source file with Info() and without AssumeFPS it reports a framerate of 29.97 FPS.
Oh wait, I used “LWLibavVideoSource”, indeed with “LSmashVideoSource” I get “1.7331 FPS”... And the length is about 4h10m, and it gets totally confused if I try to display a random frame, or parse the footage backwards. Then with AssumeFPS it seems much smoother indeed, although I still don't get why / how.
So is it specific to LSmashVideoSource, and if so what does it do that LWLibavVideoSource doesn't ? With AssumeFPS, how can it create frames which (unless I'm mistaken) aren't there to begin with, without a complex interpolation function which would almost inevitably produce artifacts ? (As you explained here five years ago, from a link which appears below, while we're doing some archeology.)

EDIT : Side question, is it wise to resize to 640x480 before uploading, considering that YouTube would resize it to that resolution anyway, possibly with a lesser quality algorithm ? Considering the quality it would be overkill to upscale it to 1280x720 (although I've been told that the bitrate allocation on YT was significantly better for so-called “HD” resolutions compared with 480p or 360p).

EDIT : Also, how can I set the audio delay with ffmpeg ? (If muxing the native audio with the re-compressed video.)

[davexnet]

What about this version?
https://youtu.be/Pc0ZHsoHAlE

[abolibibelot]

What about this version?
https://youtu.be/Pc0ZHsoHAlE

Well, downloading a video from YouTube to re-upload it on YouTube is not at all ideal, generally speaking... The video bitrate is only 351kbps (in 640x480), the framerate is 24 FPS which is unlikely to be the native one, the picture is softer than the 768x576 version.
But, as George Carlin himself used to say : “Drop some of your needs. Life is a zero-sum game.”

[jagabo]

I don't get it. I adapted my script based on your advice :

V = LWLibavVideoSource("K:\GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].mp4").BicubicResize(640,480).AssumeFPS(29.97)
A = LWLibavAudioSource("K:\GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].mp4")
AudioDub(V, A).FadeIn(20).FadeOut(20).DelayAudio(0.200)
TextSub("K:\GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].srt")

The playback is just as jerky in VDub2 and the resulting encoded file is strictly identical to the previous one made without AssumeFPS (same checksums – the audio delay did nothing because I copied the audio from the source file for the encode, considering that it doesn't require the fade-in / fade-out so it avoids a generation loss).
If I load the source file with Info() and without AssumeFPS it reports a framerate of 29.97 FPS.
Oh wait, I used “LWLibavVideoSource”, indeed with “LSmashVideoSource” I get “1.7331 FPS”... And the length is about 4h10m, and it gets totally confused if I try to display a random frame, or parse the footage backwards. Then with AssumeFPS it seems much smoother indeed, although I still don't get why / how.
So is it specific to LSmashVideoSource, and if so what does it do that LWLibavVideoSource doesn't ? With AssumeFPS, how can it create frames which (unless I'm mistaken) aren't there to begin with, without a complex interpolation function which would almost inevitably produce artifacts ? (As you explained here five years ago, from a link which appears below, while we're doing some archeology.)

EDIT : Side question, is it wise to resize to 640x480 before uploading, considering that YouTube would resize it to that resolution anyway, possibly with a lesser quality algorithm ? Considering the quality it would be overkill to upscale it to 1280x720 (although I've been told that the bitrate allocation on YT was significantly better for so-called “HD” resolutions compared with 480p or 360p).

EDIT : Also, how can I set the audio delay with ffmpeg ? (If muxing the native audio with the re-compressed video.)

LSmash uses the index that's within the MP4 or MOV file, and only works with those containers. LWlibab builds its own index (hence the .lwi file) and works with any container. Apparently both of them screw up with this video, but in different ways.

AssumeFPS() doesn't create or remove any frames. All the frames are already present in the MP4 file. ffprobe.exe shows the file contains 25985 frames. At 29.97 fps that equates to about 14 minutes and 27.033 seconds.

I don't much about what youtube does with uploads.

[abolibibelot]

Alright then. Indeed the playback of the native video is smooth in VLC Media Player (only a few duplicated frames, which are possibly not really duplicated as I can't see a single one with LSmashVideoSource + AssumeFPS).
Is there some way to detect that kind of issue right away ? Is it a common issue with variable framerate videos and Avisynth source plugins ?
Also, as I mentioned earlier, LSmashVideoSource seems to get confused when scrubbing backwards, isn't it a problem when using non-linear filters ?
MediaInfo says about the native video :

Type d'images/s                          : Variable
Images par seconde                       : 29,970 (29970/1000) Im/s
Images/s mini                            : 29,960 Im/s
Images/s maxi                            : 29,970 Im/s

So if there is a variability it's extremely small, yet enough to cause trouble.
In this thread it is said that the video in question was “made by an idiot” – could it have been a similar issue instead ?

As for what YT does, as is stated in a thread linked at the bottom of this page, it only accepts a few values for the vertical resolution, and will any uploaded video to the nearest inferior size among those, so when uploading a 576p video it gets downscaled to 480p. Otherwise, “poisondeathray” wrote a few months ago :
“YT is one of the few cases where upscaling to at least "HD" 720 height is beneficial. It allocates more bitrate in proportion, and even the SD version of HD looks better than the SD version. Usually a sharper resizer in general for upscaling, you can try different ones. Default LanczosResize is 3 tap, you can try 4 for sharper for example, but you will get ringing artifacts. It depends on the source. You can do some small tests (e.g. upload a small test section), then evaluate,change, repeat. Because practices are changing at YT all the time. H.264 seems to be going down in quality, and VP9 encodes going up in general. But it's not clear how/why YT makes VP9 available on some videos, but not others.”

Again, is there a ffmpeg switch to set an audio delay ?

[jagabo]

Is there some way to detect that kind of issue right away ?

Be suspicious of any VFR video in AviSynth. Try ffVideoSource, LSmashVideoSource, LWlibavVideoSource, and DirectShowSource. In theory DirectShowSource("filename.ext", ConvertFPS=true, FPS=??.???) will convert VFR to CFR by duplicating or decimating frames.

Is it a common issue with variable framerate videos and Avisynth source plugins ?

Yes.

Also, as I mentioned earlier, LSmashVideoSource seems to get confused when scrubbing backwards, isn't it a problem when using non-linear filters ?

Yes, it can be a problem in AviSynth (and with media players too). You can work around it by using only the source filter (hence linear access) and saving to an all I-frame intermediate. Then do your editing/filtering with that.

MediaInfo says about the native video :

Type d'images/s                          : Variable
Images par seconde                       : 29,970 (29970/1000) Im/s
Images/s mini                            : 29,960 Im/s
Images/s maxi                            : 29,970 Im/s

So if there is a variability it's extremely small, yet enough to cause trouble.

On my computer it says the min is 29.91 and the max is 100.000. Looking at ffprobe's report:

ffprobe.exe -threads %NUMBER_OF_PROCESSORS%*1.5  -v quiet -pretty -print_format compact -show_entries "frame=coded_picture_number,pkt_pts_time,pkt_duration_time,pict_type," -select_streams v:0 INPUT.MP4 >OUTPUT.TXT

frame|pkt_pts_time=0:00:00.000000|pkt_duration_time=0:00:00.033333|pict_type=I|coded_picture_number=0
frame|pkt_pts_time=0:00:00.100089|pkt_duration_time=0:00:00.033333|pict_type=B|coded_picture_number=3
frame|pkt_pts_time=0:00:00.066711|pkt_duration_time=0:00:00.033378|pict_type=B|coded_picture_number=2
frame|pkt_pts_time=0:00:00.133422|pkt_duration_time=0:00:00.033378|pict_type=B|coded_picture_number=4
frame|pkt_pts_time=0:00:00.033333|pkt_duration_time=0:00:00.033378|pict_type=P|coded_picture_number=1
frame|pkt_pts_time=0:00:00.233556|pkt_duration_time=0:00:00.033378|pict_type=B|coded_picture_number=7
frame|pkt_pts_time=0:00:00.200178|pkt_duration_time=0:00:00.033378|pict_type=B|coded_picture_number=6
frame|pkt_pts_time=0:00:00.266933|pkt_duration_time=0:00:00.033333|pict_type=B|coded_picture_number=8
frame|pkt_pts_time=0:00:00.166800|pkt_duration_time=0:00:00.033378|pict_type=P|coded_picture_number=5
...
frame|pkt_pts_time=0:14:26.708944|pkt_duration_time=0:00:00.033244|pict_type=P|coded_picture_number=25975
frame|pkt_pts_time=0:14:26.875700|pkt_duration_time=0:00:00.033422|pict_type=B|coded_picture_number=25980
frame|pkt_pts_time=0:14:26.842278|pkt_duration_time=0:00:00.033422|pict_type=B|coded_picture_number=25979
frame|pkt_pts_time=0:14:26.809033|pkt_duration_time=0:00:00.033244|pict_type=P|coded_picture_number=25978
frame|pkt_pts_time=0:14:26.909122|pkt_duration_time=0:00:00.033244|pict_type=P|coded_picture_number=25981
frame|pkt_pts_time=0:14:26.942367|pkt_duration_time=0:00:00.033422|pict_type=P|coded_picture_number=25982
frame|pkt_pts_time=0:14:26.975789|pkt_duration_time=0:00:00.033422|pict_type=P|coded_picture_number=25983
frame|pkt_pts_time=0:14:27.009211|pkt_duration_time=0:00:00.010000|pict_type=I|coded_picture_number=25984

There are a few other durations around 0.033???. Only the last frame has that 0.0100000 (100 fps). It's also unusual that the pkt_pts_time doesn't increase with every frame (ffprobe displays in presentation order). There are many backwards steps. Those are obviously wrong -- and I think that is the cause of some of the confusion. Maybe this just a muxing problem from trimming?

Again, is there a ffmpeg switch to set an audio delay ?

I'm sure there is but I don't know it off the top of my head. I'm sure you can google it as well as I...

[abolibibelot]

On my computer it says the min is 29.91 and the max is 100.000.

Then that was introduced by Avidemux when I cut that excerpt. The MediaInfo report above is based on the original full-length file. I get the same values when analysing the excerpt instead.
When loading the original file and the Avidemux cut excerpt in AVSPMod, using LWLibavVideoSource, and synchronizing them, I can clearly see that many frames which do exist in the full-length file appear as duplicates in the stuttering excerpt.

Vorig = LWLibavVideoSource("H:\[Brain Droppings]-147213.mp4").BicubicResize(640,480).Trim(19151,45135)
Vedit = LWLibavVideoSource("K:\GCarlin extrait 10m39s72-25m06s72 [Avidemux MP4v2 Muxer].mp4").BicubicResize(640,480)
StackHorizontal(Vorig,Vedit)

So... is this a known issue with Avidemux 2.6, when using the “MP4v2 Muxer” ?
(If someone wants to download the full-length video from C-Span : youtube-dl https://www.c-span.org/video/?123268-1/brain-droppings – it automatically selects the best quality out of three available.)

Maybe this just a muxing problem from trimming?

It would seem like it. But I don't know if it's somehow due to something somewhat irregular in the source (full-length) video (although it seems to play fine everywhere and to be correctly loaded by LWLibavVideoSource), or if it's a general problem with Avidemux's MP4 muxer.




-------------------

I'm sure there is but I don't know it off the top of my head. I'm sure you can google it as well as I...

Indeed but :
– I had already tried to find such a command in ffmpeg's manual, searching the huge text file with the key-word “delay” which brought many hits but nothing relevant.
– Being already in a situation involving “unknown unknowns” I figured that it would be wise to ask for a tip or two on that aspect too...
It turns out that it's indeed doable but quite convoluted (I guess that it's due to how versatile that tool is). I was lucky enough to read first this article (although it came 8th in Google's results), which provides a clear and thorough guide on that specific topic :
https://wjwoodrow.wordpress.com/2013/02/04/correcting-for-audiovideo-sync-issues-with-...offset-switch/
It says in its introduction, echoing what I wrote above :

The ffmpeg program has numerous “switches” that help to adjust and convert audio and video files. Some of them are not explained very well in the documentation, and many websites have confusing postings by well-meaning people trying to make use of the switches.

Case in point, I then read this VideoHelp thread (from 2012), which is a tad more confusing :
https://forum.videohelp.com/threads/346293-Insert-Audio-Delay-With-ffmpeg
In particular someone said

Caveats :
- you should be using a recent version of ffmpeg as the syntax has changed over time.
- itsoffset doesn't seem to support negative values, so use one of the examples above.

But then :

When copying the audio without transcoding, my examples in post #4 are correct. But when re-encoding the audio, the ffmpeg formatting needs to be different.
For the latter, the mapping options stay the same, and the itsoffset parameter does allow negative values... I overlooked this as I didn't try all possible combinations of delaying the audio/video and coping/transcoding.
These should work when converting the audio:
delays audio:

ffmpeg -i test.avi -itsoffset -1 -i test.avi -map 1:0 -map 0:1 -vcodec libx264 -r 25 -b:v 1268k -s 512x368 -acodec libmp3lame audio_delayed.mp4

delays video:

ffmpeg -i test.avi -itsoffset 1 -i test.avi -map 1:0 -map 0:1 -vcodec libx264 -r 25 -b:v 1268k -s 512x368 -acodec libmp3lame video_delayed.mp4

[...]
Sorry if I've caused you an evening of banging your head on your desk

Also intersting :
https://superuser.com/questions/982342/in-ffmpeg-how-to-delay-only-the-audio-of-a-mp4-...-the-au/983153
Which leads to :
https://alien.slackbook.org/blog/fixing-audio-sync-with-ffmpeg/
The 2011 article is about as clear as the first linked above, but in the comments I found this exchange, which exposes some more caveats :

– Trying to fix audio sync for an automated video recording workflow. -itoffset is working great, but the effect is only seen when playing back the files in a player. If I bring the output files into an editing program like Premiere, the audio/video tracks are unchanged. Is -itoffset just modifying the playback timestamp for the audio/video tracks? These files are headed to video editors downstream in the workflow, so I’m trying to get a fix that fixes the actual file, not just modifies metadata.
– Interesting observation. Looks like this is an issue which is specific to MP4 containers, as there’s an old bug which describes the issue you are experiencing: https://trac.ffmpeg.org/ticket/1349
– I had a similar issue last year when working with files created by an Android app – that time caused by wildly variable framerates that averaged out to 24fps over enough time, but had the same symptom here. Bring the mp4 into After Effects for post-processing, and suddenly audio is out of sync. Solution that time was to transcode to a mezzanine codec first (ProRes) which magically read the PTS of the mp4 and gave a synced file that could be used in AfterEffects. My issue this time is delay that is caused by the physical workflow in the recording setup (hardware is adding a couple frames video delay, and the encoder is adding a couple more adding up to about 6 frames of video delay. It’s looking to me like there isn’t a one-line quick-fix here, and that I’ll need to de-mux the A and V streams, delay the audio, and mux it all back together.
– I think I have this sorted now.

ffmpeg -i source.mp4 -i source.mp4 -filter_complex “adelay=150|150” -map 0:0 -map 1:1 -c:v copy -c:a aac output-0150.mp4

Happily the video stream can just be copied over – that was going to be a big performance hit to re-encode that. So far in some limited tests this is working well and Premiere is showing correctly shifted audio. I’ll note that this solution probably would not work if audio was behind the video (needing to delay the video). That would probably require re-encoding on the video, both to accomplish the trim (probably using a ss seek?) as well as to be able to have that seek work on a sub-GOP duration.
– Alas, somehow and contrary to the man page, ffmpeg doesn’t seem to want to accept negative numbers for itsoffset. At least here it doesn’t seem to. So the delay for the video signals needs to come (almost illogically early) in front of the first stream:

ffmpeg -itsoffset 1.0 -i infile.mp4 -i infile.mp4 -map 0:0 -map 1:1 -acodec copy -vcodec copy synced_file.mp4

for a delay of 1.0 seconds.

-------------------

I find it quite difficult to identify precisely the audio delay. I didn't even notice that there was one until you mentioned it. Do you have a specific method for that or is it just by experience that you hear it right away even when there's such a small delay ? (Which might actually not be in the source file, I'm struggling to compare the two. Mark Twain wrote that “you can't depend on your eyes when your imagination is out of focus”, but that's true for the ears as well !)

[poisondeathray]

I find it quite difficult to identify precisely the audio delay. I didn't even notice that there was one until you mentioned it. Do you have a specific method for that or is it just by experience that you hear it right away even when there's such a small delay ? (Which might actually not be in the source file, I'm struggling to compare the two. Mark Twain wrote that “you can't depend on your eyes when your imagination is out of focus”, but that's true for the ears as well !)

You can use a visual aid such as an audio waveform in a video editor to note the relation to the video.

Avisynth has a few too, that will probably be good enough for the job

waveform
https://forum.doom9.org/showthread.php?t=165703
audiograph
http://avisynth.nl/index.php/AudioGraph

(IMO a NLE is nicer because playback is better optimized, you can dynamically zoom into waveform etc...)


BUT if you are venturing into VFR land , and it's significantly variable - and if this was a CFR conversion somewhere - there are going to be areas where the audio delay value will change in different sections. You might choose some constant + or - offset, but it will never be "perfect" everywhere as if was the original CFR recording. (Or was the VFR business just from avidemux and remuxing ? sorry I didn't read the whole thread in detail)

[jagabo]

Yes, I've used AudioGraph() to check A/V sync. But in this case I just eyeballed it to get a rough match.

[abolibibelot]

And so, is Avidemux known to be problematic when it comes to trimming MP4 files (with no transcoding) ? Or is it a problem with version 2.6 in particular ?
What is the preferred method for this, mp4box perhaps ? I use it regularly for muxing or extracting an audio stream, but for trimming it's more convenient to see the display simultaneously and jump from key-frame to key-frame to know precisely where it will be cut. If it's really more reliable I could use Avidemux to determine the timings of the cutting points, and then run a mp4box trimming command with those timings – assuming that they're not off, as I found out to be the case between Avidemux and MKVToolNix (when trimming the samples for this thread, after several trials and errors, I had to set a start timing 7 seconds earlier with MKVToolNix compared with Avidemux – probably at the previous key-frame – to get the same actual begining).