Video Freezing Issue at the Start of Clips Cut with ffmpeg

[doganjr]

Hi everyone,

I'm working on extracting 6-7 second clips from a long .mp4 video (about 2.5 hours) and then combining these clips. I'm aiming for the process to be as fast as possible.

So far, I've managed to cut and merge the clips using the ffmpeg library, and it has been impressively fast—just as I hoped.

However, the segments I cut with ffmpeg have an issue: for the first second or so, the audio plays, but the video is frozen. I've uploaded an example segment to Google Drive and shared the link below:

https://drive.google.com/file/d/1-_NLoEtZjrW6ad6KJPYnM3EJC5QOnTr2/view?usp=sharing

I came across some information online about keyframes, but I’m not very knowledgeable about this topic.

I'm doing the cutting and merging using Python, and I’ve shared the function I’m using below.

def cut_and_merge_with_ffmpeg(video_path, frame_indices, output_path, frame_offset=500, fps=60, temp_dir="temp_segments"):
    os.makedirs(temp_dir, exist_ok=True)
    segment_files = []

    for i, frame_index in enumerate(frame_indices):
        print(i)
        start_frame = max(0, frame_index - frame_offset)
        start_time = start_frame / fps 
        end_time = frame_index / fps  

        segment_file = os.path.join(temp_dir, f"segment_{i}.mp4")
        segment_files.append(segment_file)

        cmd = [
            "ffmpeg",
            "-y", 
            "-i", video_path,  
            "-ss", f"{start_time:.6f}",  
            "-to", f"{end_time:.6f}",  
            "-vcodec", "copy",  
            "-acodec", "copy",
            segment_file
        ]
        subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

    concat_file = os.path.join(temp_dir, "concat_list.txt")
    with open(concat_file, "w") as f:
        for segment_file in segment_files:
            f.write(f"file '{os.path.abspath(segment_file)}'\n")

    cmd = [
        "ffmpeg",
        "-y",  
        "-f", "concat",  
        "-safe", "0",  
        "-i", concat_file,  
        "-c", "copy",  
        output_path
    ]
    subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)


    for segment_file in segment_files:
        os.remove(segment_file)
    os.remove(concat_file)
    os.rmdir(temp_dir)

My question is: how can I fix this freezing issue at the beginning of the clips? Is there another approach you’d recommend?

Thanks in advance!

[doganjr]

As an update, I changed the order of the -ss, --to and -i arguments as follows:

        cmd = [
            "ffmpeg",
            "-y",  
            "-ss", f"{start_time:.6f}",
            "-to", f"{end_time:.6f}", 
            "-i", video_path, 
            "-vcodec", "copy",  
            "-acodec", "copy",
            segment_file
        ]

This time, the audio disappeared:

https://drive.google.com/file/d/1NHEhIpUVqoL1UhRL_mCr7PQPV77RWYry/view?usp=sharing

[pcspeak]

I have suffered a similar problem on a number of occasions. For me, the solution is to make an interim video where every frame is a key frame (gop).

 ffmpeg -i "my-video.mkv" -c:v libx264 -profile:v high -level:v 4.0 -preset fast -crf 10 -g 1 -c:a copy -y "my-video-temp.mkv"

The solution has not been perfect, but has covered 95% of my needs.
I find I need to adjust the parameters for ffmpeg on each occasion. Usually just -crf 10. Maybe to -crf 20. I look for a rate factor giving the output approx. the same bit rate as the input.

The above code gives me a ginormous output file.
Edit: Changing the fast preset to medium or slow will also make a difference to the quality of the interim video. Mostly I use the medium preset with a -crf of 21.
Cheers.

[ProWo]

With -copy you can cut on keyframes only.
Forget the commandline and use a GUI like Avidemux.

[_Al_]

https://www.ffmpeg.org/ffmpeg-formats.html#segment_002c-stream_005fsegment_002c-ssegment

Checking out docs and you need to copy on I frames, perhaps using -f segment flag and perhaps -segment_frames flag as well or -segment_time_delta,

there is different flavors for possible command line examples:
https://www.ffmpeg.org/ffmpeg-formats.html#Examples-15

[_Al_]

So idea, you can very quickly create segmented files with no re-encoding, and then using python using created csv file list to use paths only for concat,
to create a segment every 10000 frames (166 seconds if 60fps)

ffmpeg -i INPUT -codec copy -map 0 -f segment -reset_timestamps 1 -segment_list out.csv -segment_frames 1000, 10000, 20000  out%03d.mp4

get every tenth file only from segmented list

TAKE_EVERY_NTH_FILE = 10
segment_paths = []
with open("out.csv") as f:
    for row in f:
        segment_paths.append(row.split(",")[0])
print(segment_paths)

concat_paths = segment_paths[::TAKE_EVERY_NTH_FILE]
print(concat_paths)

it might be possible doing right away, segmented files with gaps in video, but not sure how

[_Al_]

Still cannot come up with something that can create segments needed only, but this is really pretty fast also, as mentioned, it creates all segments, not encoding anything and then selecting every n-th segment for a final concat:

from pathlib import Path
import subprocess
import shlex

FFMPEG = r"C:\tools\ffmpeg.exe"
FFMPEG = Path(FFMPEG).as_posix()

def process(cmd):
    cmd = shlex.split(cmd)
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
    r = p.communicate()
    if p.returncode:
        #print error if there was
        print(r[1])    

def clean_dir(directory: Path):
    #remove all files in a directory
    [f.unlink() for f in directory.glob("*") if f.is_file()] 

def cut_and_merge_with_ffmpeg(video_path, output_path, temp_dir, segment_time, take_every_nth_segment):
    temp_dir = Path(temp_dir)
    temp_dir.mkdir(parents=True, exist_ok=True)
    clean_dir(temp_dir) 
    video_path = Path(video_path).as_posix() # make a string with proper slashes so shlex work
    output_path = Path(output_path).as_posix()

    #get segments
    segment_outputs = temp_dir / "out%04d.mp4"
    segment_outputs = segment_outputs.as_posix()
    cmd = f'"{FFMPEG}" -i "{video_path}" -map 0:v:0 -map 0:a:0 -c copy '\
          f'-f segment -reset_timestamps 1 -segment_time {segment_time} -segment_list out.csv "{segment_outputs}"'
    process(cmd)

    #get concat list from segments
    segment_files = []
    with open("out.csv") as f:
        for row in f:
            segment_files.append(row.split(",")[0])
    concat_files = segment_files[::take_every_nth_segment]
    concat_file = temp_dir / "concat_list.txt"
    concat_file = concat_file.as_posix() # make a string from a Path object
    with open(concat_file, "w") as f:
        for file in concat_files:
            f.write(f"file '{temp_dir / Path(file).name}'\n")

    #ffmpeg concat
    cmd = f'"{FFMPEG}" -f concat -safe 0 -y -i "{concat_file}" -map 0:v:0 -map 0:a:0 -c copy "{output_path}"'
    process(cmd)

#usage:
video_path = r"E:\video.mp4"
output_path = r"F:\OUT.mp4"
temp_dir = r"E:\temp" #directory will be cleaned up!, use new path for this purpose
segment_time = 10 #seconds
take_every_nth_segment = 70

cut_and_merge_with_ffmpeg(video_path, output_path, temp_dir, segment_time, take_every_nth_segment)
clean_dir(Path(temp_dir))

[videoAI]

LosslessCut

[jagabo]

In copy mode ffmpeg (or any other editor) can only cut video on keyframes. The nearest keyframe to your cut point was about 1.5 seconds after the specified cut-in time. The nearest audio frame was 27 milliseconds after the cut-in point. So when you play the A/V file about 1.5 seconds of audio plays before the video starts playing (the first frame of video is displayed for that 1.5 seconds).

[_Al_]

There is a way to find out keyframe positions for a video using ffprobe, but it takes considerable amount of time. If that was a concern. Especially using python, it is comfortable to do that. Maybe for op would be enogh to come up with second keyframe only (to start slicing there) if rest of those segment cuts turn out ok (ffmpeg lands it on nearby keyframes for some reason).

[jagabo]

Here's an ffprobe report of the start of the file in post #2:


[Attachment 83846 - Click to enlarge]

You can see that the file starts an audio packet, line 2, with a timestamp of 0.027 seconds. It is followed by many other audio packets. The first video packet is at line 37 and has a timestamp of 1.50067 seconds. So the media player starts playing the the audio then starts playing the video 1.5 seconds later. The exact behavior depends on the player. Some will show a black screen for the first 1.5 seconds. Some will show the first frame of video for that duration. Some may start playing the video right away but skip the first 1.5 seconds of audio.

[_Al_]

Yes, I meant to investigate the original file by ffprobe. The posted video is not original and it is a result of landing first slice point off the keyframe. So ffmpeg while muxing comes up with something like that , frozen frame, or something else. What I could read about it does not search for nearby keyframe and cuts it there. Instead it does something like that in that posted video.

I assume op deals with many videos and whants some en masse solution, for many videos, not gui, possibly fast solution.

What I gathered so far:
I found in docs that -f segment feature, that segments video actually makes sure, it cuts segments only on keyframes. Tested it myself, and videos are segmented nicely. If for example, I used -f segment -reset_timestamps 1 -segment_time 10, it could come up with segments length not exactly 10 seconds, but close, because it makes sure it always cuts on keyframes. It depends on video. If there were no keyframes, it would not even segmented video.

Using -ss start_time -to end_time method cannot guarantee to cut on keyframe. That is what I got from web. Different workarounds were used to come up with a solution to cut on keyframe, but it always involves knowing actual keyframe point in the first place. And that is what takes really a time. Using ffprobe and original file. Just that probing takes a multiple of times more what segment method alone would take. Segment method using -f segment always cuts only on keyframes. So I attempted to use it and then join desired segments that were created. I was surprised how fast it actually was. Basically it takes time only to physically write segments to disk. Then second ffmpeg command,the actual concat command of desired segments is even lightning fast, because it joins only fraction of those segments together and writes it on a disk. If input video, temp_directory and final output are spread on different disks, not system disk, it is pretty fast.

segment can be created by:
-f segment -reset_timestamps 1 -segment_time segment_length
where in our case to reset timestamps is needed, but in case of just playing all segments one after another for other purposes (internet download, mostly in the past) it was actually handy not to reset timestamps
or
-f segment -reset_timestamps 1 -segment_frames 100,8000,9000,...
if frames are desired for cuts, but again, those frame numbers are approximate at the end, because a closest keyframe is taken instead

[doganjr]

I have suffered a similar problem on a number of occasions. For me, the solution is to make an interim video where every frame is a key frame (gop).

 ffmpeg -i "my-video.mkv" -c:v libx264 -profile:v high -level:v 4.0 -preset fast -crf 10 -g 1 -c:a copy -y "my-video-temp.mkv"

The solution has not been perfect, but has covered 95% of my needs.
I find I need to adjust the parameters for ffmpeg on each occasion. Usually just -crf 10. Maybe to -crf 20. I look for a rate factor giving the output approx. the same bit rate as the input.

The above code gives me a ginormous output file.
Edit: Changing the fast preset to medium or slow will also make a difference to the quality of the interim video. Mostly I use the medium preset with a -crf of 21.
Cheers.

Thanks for the reply!

I changed the command as follows and it solved most of the problems that itches me. I know it is not the perfect scenario that I want but decent:

    cmd = [
        "ffmpeg",
        "-y",  
        "-f", "concat",  
        "-safe", "0", 
        "-i", concat_file, 
        "-c", "copy",  
        output_path
    ]

[doganjr]

With -copy you can cut on keyframes only.
Forget the commandline and use a GUI like Avidemux.

Unfortunately, I cannot!

It is a 3 hour video, which will be very painful to edit without using code

[doganjr]

Still cannot come up with something that can create segments needed only, but this is really pretty fast also, as mentioned, it creates all segments, not encoding anything and then selecting every n-th segment for a final concat:

from pathlib import Path
import subprocess
import shlex

FFMPEG = r"C:\tools\ffmpeg.exe"
FFMPEG = Path(FFMPEG).as_posix()

def process(cmd):
    cmd = shlex.split(cmd)
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
    r = p.communicate()
    if p.returncode:
        #print error if there was
        print(r[1])    

def clean_dir(directory: Path):
    #remove all files in a directory
    [f.unlink() for f in directory.glob("*") if f.is_file()] 

def cut_and_merge_with_ffmpeg(video_path, output_path, temp_dir, segment_time, take_every_nth_segment):
    temp_dir = Path(temp_dir)
    temp_dir.mkdir(parents=True, exist_ok=True)
    clean_dir(temp_dir) 
    video_path = Path(video_path).as_posix() # make a string with proper slashes so shlex work
    output_path = Path(output_path).as_posix()

    #get segments
    segment_outputs = temp_dir / "out%04d.mp4"
    segment_outputs = segment_outputs.as_posix()
    cmd = f'"{FFMPEG}" -i "{video_path}" -map 0:v:0 -map 0:a:0 -c copy '\
          f'-f segment -reset_timestamps 1 -segment_time {segment_time} -segment_list out.csv "{segment_outputs}"'
    process(cmd)

    #get concat list from segments
    segment_files = []
    with open("out.csv") as f:
        for row in f:
            segment_files.append(row.split(",")[0])
    concat_files = segment_files[::take_every_nth_segment]
    concat_file = temp_dir / "concat_list.txt"
    concat_file = concat_file.as_posix() # make a string from a Path object
    with open(concat_file, "w") as f:
        for file in concat_files:
            f.write(f"file '{temp_dir / Path(file).name}'\n")

    #ffmpeg concat
    cmd = f'"{FFMPEG}" -f concat -safe 0 -y -i "{concat_file}" -map 0:v:0 -map 0:a:0 -c copy "{output_path}"'
    process(cmd)

#usage:
video_path = r"E:\video.mp4"
output_path = r"F:\OUT.mp4"
temp_dir = r"E:\temp" #directory will be cleaned up!, use new path for this purpose
segment_time = 10 #seconds
take_every_nth_segment = 70

cut_and_merge_with_ffmpeg(video_path, output_path, temp_dir, segment_time, take_every_nth_segment)
clean_dir(Path(temp_dir))

Hmm, thanks for the reply! Interesting approach

[doganjr]

LosslessCut

It is such a quite good software! However, there will be total number of approximately 90-100 segments, each would last 8-9 seconds. Also, I need to cut and merge these from a relatively long video. So, GUI will be very precise, I agree, however, It is not affordable in terms of time and workload

[doganjr]

Here's an ffprobe report of the start of the file in post #2:


[Attachment 83846 - Click to enlarge]

You can see that the file starts an audio packet, line 2, with a timestamp of 0.027 seconds. It is followed by many other audio packets. The first video packet is at line 37 and has a timestamp of 1.50067 seconds. So the media player starts playing the the audio then starts playing the video 1.5 seconds later. The exact behavior depends on the player. Some will show a black screen for the first 1.5 seconds. Some will show the first frame of video for that duration. Some may start playing the video right away but skip the first 1.5 seconds of audio.

It is great to see so wise and helpful people under my thread. These type of posts widen my vision! Thanks for the reply!