Upgraded Ubuntu from 20.04.6 to 24.04.1 and ffmpeg slowed down a lot!

[BosseB]

I have an Ubuntu server, which I use for various things among which are video downloading of news shows.
The download is done using a shell-script calling ffmpeg and has been working fine for years.
Then I also use ffmpeg to cut-and-paste the videos to remove ads from them before I view them.
This is done by viewing the raw video in a player I programmed on Windows that allows me to navigate the video and set precise cut points for the video sections to keep.
The output of this is a command line argument for the cuts that call a native Ubuntu program I also wrote long time ago which assembles a sequence of ffmpeg calls to extract sections from the video into smaller parts between the cut points and finally an ffmpeg call to paste them all together again.

Now it turns out that after the Ubuntu upgrade which probably brought a newer ffmpeg (now at version 6.1.1) the extraction-paste sequence takes a much longer time to perform.

Today I timed it to get a read of the actual time for the split/combine operation:
The source video is 2h05m long and there are 10 cuts totaling 1h23m and the conversion normally took about 9s on my earlier Ubuntu version 20.04.6 and now it took 74s which is an increase of some x8!

The problem may arise in two places:

1) The download script ffmpeg command which is designed to create an mp4 video with internal "chunks" set on even seconds and now is maybe no longer doing that properly?

2) The ffmpeg commands to do the cut/pasting might no longer work as smoothly as before.

To check this I have a machine running Linux Mint from about the same age and it too downloads the videos using the same command.
It has ffmpeg version 4.2.7 so I used the same command on that to cut-and-combine the same video using that computer and it was done in 9.5 seconds!

So I made another test to check if the download or the cut/paste operation is to blame:

I transferred the video file from the LinuxMint machine to the Ubuntu server and then ran the split/combine program on that video file and it came out as 70 seconds too! So it seems like it is not the download format that is being the cause.

Then I also transferred the video file from the server to the LinuxMint machine and ran the split/combine command on it there.
This turns out to result in a processing time of 9.2 seconds!
This also proves to me that the download itself is not doing bad things to the file.

So I think that the video itself is not compromised.

End result:
1) The video downloaded on both machines are both split and combined in the same quick time of about 9s on the LinuxMint machine.

2) The split/combine done on the upgraded server takes between 7 times longer than on the LinuxMint machine.

So the split/combine operation is much slower than before the upgrade.

This is how that is being done:

The splitting uses the following command to create a number of separate smaller video files:

ffmpeg -ss starttime -i input.mp4 -to endtime -c copy output.mp4 -hide_banner

with starttime and endtime in integer seconds.

and where the output.mp4 file is created as a numbered sequence of files.

The names of these files are added to a list file of the files to be used in the combine command:

The combine itself is done using this command:

ffmpeg -f concat -i inputlist.txt -c copy outputfile.mp4

And this is all packaged into a small program that accepts an argument list of the input file name, the cuts time specifications and the output file name.

It has been working fine for years and now with the new ffmpeg version has slowed down very much.

I have observed the working directory during the process and it seems like the culprit is the extraction of the cut sections, which now takes a very long time compared to earlier.
Once all 10 cuts have been extracted the combination into one file is done in a second or two!

WHY? The only difference seems to be the ffmpeg version number...

And how can it be fixed?

[Selur]

Assuming it's not some file access time or other stuff that changed on the system.

A lot changed between FFmpeg 4.2.7 and 6.1.1 (current is 7.1...).
https://github.com/FFmpeg/FFmpeg/blob/master/Changelog
first thing I saw:

version 6.0:
...
- ffmpeg now requires threading to be built
- ffmpeg now runs every muxer in a separate thread
...

First thing I would suggest, is to use 'combined seeking' ('-ss' before and after the input '-i'), see: https://trac.ffmpeg.org/wiki/Seeking
Additionally, looking at what is happening using the debug log level might show some hints if additional stuff is done.

If your ffmpeg version were a lot older:

-ss position (input/output)

When used as an input option (before -i), seeks in this input file to position. Note that in most formats it is not possible to seek exactly, so ffmpeg will seek to the closest seek point before position. When transcoding and -accurate_seek is enabled (the default), this extra segment between the seek point and position will be decoded and discarded. When doing stream copy or when -noaccurate_seek is used, it will be preserved.

When used as an output option (before an output url), decodes but discards input until the timestamps reach position.

position must be a time duration specification, see (ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual.

source: https://ffmpeg.org/ffmpeg-all.html#Main-options
+

-accurate_seek (input)

This option enables or disables accurate seeking in input files with the -ss option. It is enabled by default, so seeking is accurate when transcoding. Use -noaccurate_seek to disable it, which may be useful e.g. when copying some streams and transcoding the others.

source: https://ffmpeg.org/ffmpeg-all.html#Advanced-options
might be the cause of the option. Before 2.1 '-noaccurate_seek' was the default, now its '-accurate_seek'.

WHY? The only difference seems to be the ffmpeg version number...

Unless you tested different ffmpeg version on the same system, a lot changed between Ubuntu 20.04 and 24.04.

Cu Selur

[BosseB]

So I had a look at the actual sourcecode for my videosplitcombine program to locate the exact commands issued when processing the input file.

1) The extraction phase creating the cutout video sections

This is using the following command to extract a section given the start point and duration in integer seconds for each cut given as arguments to the program.
It loops all start/duration pairs and creates a cut video file for each with a sequential name, which is also written into a joinfilelist.txt file.

ffmpeg -hide_banner -i <inputfile> -ss <start> -t <duration> -c:v copy -c:a copy <outputfile>

2) The concatenation phase to combine the video cuts

Here the created video cuts are listed by name on the disk in the joinfilelist.txt and the command looks like this:

ffmpeg -hide_banner -f concat -i joinfilelist.txt -c copy <outputfilename>

When I time this process I can see that the creation of the cuts is what takes this very long time in Ubuntu 24.04.1 compared to 20.04.6 on the exact same hardware (I dist-upgraded the Ubuntu installation).

Once the cuts are all done the final concat runs pretty speedily, like in a second or so for a 1 hour video with 10 cuts.

[Selur]

So getting to the required position is probably what takes long.
Issue could either be that the decoding changed in the versions (higher loglevel should hint to this) or something with the file access changed (unlikely unless the files are not local).

[BosseB]

So getting to the required position is probably what takes long.
Issue could either be that the decoding changed in the versions (higher loglevel should hint to this) or something with the file access changed (unlikely unless the files are not local).

FORENSIC REPORT:

I have now tested manual conversion using the ffmpeg commands directly on the command line as shown below to convert a 1h video file to a 41m45s output video:

1) I ran this repeatedly using the extracted cut points as -ss and -to arguments and incremented x in cutx.mp4 sequentially to get a set of 6 cut files.

time ffmpeg -ss 1050 -i Input.mp4 -to 464 -c copy cutx.mp4 -hide_banner

The sum of the reported times was: 4.671s


2) Then create a joinfiles.txt containing this:

file 'cut1.mp4'
file 'cut2.mp4'
file 'cut3.mp4'
file 'cut4.mp4'
file 'cut5.mp4'
file 'cut6.mp4'

3) Then finally run this concatenation command:

time ffmpeg -f concat -i joinfiles.txt -c copy Output.mp4 -hide_banner

The reported time for this test was: 3.634s

The sum of the processing times came out as 8.3 seconds which is what I had expected from it before the Ubuntu release upgrade.

So in fact the current version of ffmpeg is not slowing things down if the calls are made properly.

So I have scrutinized the sourcecode of my converter program and found a possible cause of the problem in a "fix" that was put in the program earlier when ffmpeg had problems finding the cut starts.

It seems like the sourcecode shifts the first cut point by a fraction of a second and this in turn makes ffmpeg start scanning for the start point for each of the cuts being extracted. That can easily add a few seconds and this multiplied by the number of cuts to make could be quite a long delay.

Unfortunately I cannot reach the development IDE for the executable right now and change it because that is on a headless Desktop Ubuntu machine which I usually reach using VNC, but following some updates a few months ago it now errors out when trying to start the TigerVNC server on that machine.....

But I can see the source-code via SSH and there I found this suspicious start shift...

I might as well try to modify my Windows video editor program that now generates the call to the splitcombine program to instead generate the direct ffmpeg calls. Or else write a shell-script program that can run on Linux doing the same thing.

[Selur]

Happy, you found the problem.

[BosseB]

Right, and I also found out how I could rebuild my tool from the command line on the Ubuntu machine with non-working VNC!
So I have fixed the bug and rebuilt the tool and moved it over to the Ubuntu server where it now works as I wanted it to.
Now back to process times between 8 and 20 seconds for 1-2 hour videos with 5 cuts per hour.

[BosseB]

Now tested on a number of new videos and it seems like the new version of ffmpeg is still slower than the old one even though it is not by such a large factor as before the change I did as described above....

It sits at 22-28 seconds for 1-hour shows being cut down to 41 min with 5-6 segments.
Used to be 8-9 seconds...
And the thing that takes the extra time is extracting the cuts from the original video.
Concatenation takes 1-2 seconds.

[BosseB]

It sits at 22-28 seconds for 1-hour shows being cut down to 41 min with 5-6 segments.
Used to be 8-9 seconds...

Don't know why but it is now back down to 10-12 s per 1h video conversion....
Might have been something else capturing CPU power at the tests above.

[Selur]

Happy, that solved itself, and in the future you might want to check what's running in the background.

Cu Selur