How to join a great quantity of clips?

[vmackey]

Hello,

I have a big quantity of mini clips (10 to 12 frames) to join together to create one single video, and the amount of clips is around 50,000. I have tried many solutions, ffmpeg, regular editors (Davinci, Premiere), etc... but they can't handle the quantity. Can you advise me of a good solution that would help me joining that much clips without too much issue (Software with GUI preferable please)? Thanks.

[jagabo]

Although I've never concatenated 50000 clips with ffmpeg I suspect it capable of doing so. But you need to build a text file with a list of all the files:

file 'file00001.mp4'  
file 'file00002.mp4'  
...
file 'file50000.mp4'

and use a command like:

ffmpeg -y -safe 0 -f concat -i list.txt -c copy output.mp4

You can use the CLI command "DIR /B *.mp4" > list.txt" to create a list. It will have only the bare filenames so you'll have to use Notepad or some other text editor to add the preceding "file '" and ending "'" (easily done with a global replace). If your files aren't mp4 files just replace all the "mp4" with the correct extension.

[Cornucopia]

"... without too much issue." There's the rub.

1st, and most important issue is clip format similarity - resolution, framerate, codec, GOP composition...
Whoops, that's more than one issue.
Next, player gracefulness when encountering dissimilar clip transitions
Next, not all codecs will be accepted into the mp4 container.


Scott

[pcspeak]

To go with jagabo's idea.

for %%a in (*.mp4) do echo file '%%a' >> list.txt
ffmpeg -safe 0 -f concat -i list.txt -c copy -bsf:v h264_mp4toannexb Output.mp4

The highlighted parameter is what I use. It may not be needed.

Cornucopia is right. Things will probably go amiss.
You may need to do an encode of them all first. I would, to a .ts format.

[_Al_]

That ffmpeg might work of course with some prep.
Vapoursynth could be used if clips need to be converted to the same format.
The only downside is that total number of frames are needed beforehand, final video length, or just overshoot that total, but black frames will be padded.

Frames are generated on the fly basically, on need to get basis, RAM does not get bloated:

#python code, not php
import vapoursynth as vs
from vapoursynth import core
import os

DIRECTORY = r'D:\directory_with_many_files'    #directory where sources are
EXT   =  "mp4"          #source extensions
WIDTH =  1920
HEIGHT = 1080
LENGTH = 500000     #total number of frames for whole video
FPSNUM = 25
FPSDEN = 1
FORMAT = vs.YUV420P8    #or vs.RGB24 for 8bit rgb, depending on your source
SOURCE_PLUGIN = core.lsmas.LibavSMASHSource  #depends what your source is

def get_clips():
    for filename in os.listdir(DIRECTORY):
        if filename.endswith(EXT):
            '''
            here could be clip converted to the same format
            or different source plugins could be used based on source extension etc.
            '''
            path = os.path.join(DIRECTORY, filename)
            clip = SOURCE_PLUGIN(path)
            yield clip

class Clip_handler:
    def __init__(self, get_clips):
        self.clip_generator = get_clips()
        self.clip_length = 0
        self.clip_frame = -1
        self.clip = 1

    def load_new_clip(self):
        try:
            self.clip = next(self.clip_generator)
        except StopIteration:
            self.clip = None
        else:
            self.clip_length = self.clip.num_frames
            self.clip_frame = 0
            
    def get_frame(self):
        if self.clip is None:
            return self.f.copy()
        else:
            return self.clip.get_frame(self.clip_frame).copy()
        
    def fetch_frame(self, n, f):
        self.f = f
        self.clip_frame += 1
        if self.clip_length == 0:
            self.load_new_clip()
        self.clip_length  -= 1
        return self.get_frame()
    
placeholder_clip = core.std.BlankClip(width=WIDTH, height=HEIGHT, format=FORMAT, length= LENGTH)
placeholder_clip = core.std.AssumeFPS(placeholder_clip, fpsnum = FPSNUM, fpsden = FPSDEN)
clip_handler = Clip_handler(get_clips)
joined_clips = core.std.ModifyFrame(clip=placeholder_clip, clips=placeholder_clip, selector=clip_handler.fetch_frame)
joined_clips.set_output() 

[pcspeak]

This would be my first attempt, first creating .ts format files, all with the same video and audio attributes.

All the things said by Cornucopia need to be taken into account for the first ffmpeg pass.
I included a few parameters. Modify or delete or add to as required.
I'd not be worried about the size of the final mp4 until you get a workable output.
The trick is to NOT lose any quality when creating the mp4.

-bsf:v h264_mp4toannexb - cannot be used for .ts input streams. So it's not needed.

for %%a in (*.mp4) do ffmpeg -i "%%a" -vf scale=640:360 -c:v mpeg2video -qscale:v 2 -r 30 -c:a mp2 -b:a 192k "%%~na.ts"

for %%a in (*.ts) do echo file '%%a' >> list.txt
ffmpeg -safe 0 -f concat -i list.txt -c copy Output.mp4

Cheers.