Joining a large number of files

[ElusiveProcessing]

Hey everyone! New here with registering, although this site has been super helpful to me for much longer. I just now decided to get registered because I've finally hit a problem that I cannot find a complete solution for on my own.

So what I got is 1460 .ts files (yes, that many). What I'm stuck on is... how do I join that many files? I've used MP4Box in the past to join videos, but it's always been only two that needed joining. It's easy to join them when it's only two. But with the number of files I got this time around, there's no way I can type out every single file in a command. So I tried YAMB since that allows you to join many files more easily, but it gives me an error:

[03:53:18] : Error appending E:\(filepath)\(filepath)\segment-0-converted.mp4: Bad
[03:53:18] : Joining failed.

I've also tried MP4Joiner but it's giving an error that simply says "failure", and I checked the past support tickets on that program and someone had the exact same issue. The way they solved it was removing chapter markers from the video files.

As well, Avidemux would've been an option if it wasn't for that it seems you have to append each file one by one (from what I could see anyways).

So basically I'm hoping you guys can help me with one of three solutions:
1. A way to write a cat command in MP4Box to act as a range (eg: instead of writing out file1.mp4, file2.mp4, etc... write something like file1.mp4:file1460.mp4).
2. Remove chapters from all files and run them through MP4Joiner again, but of course this method will require being able to remove the chapters in bulk as there's too many files to do anything one by one.
3. Or just something else entirely that I haven't thought of...

One other thing I need to figure out is what's the best way to convert to .mp4 without losing any quality or audio/video sync, because I tried VLC but it causes random audio cuts to happen in the converted files.

Anyways, I'll really appreciate any help or pointers!

[butterw]

It should be possible with a ffmpeg concat file list.

from https://avidemux.org/smif/index.php?msg=90928:
(for %%G in (file*.ts) do @echo file '%%G') >> journey_list.txt
ffmpeg -f concat -safe 0 -i journey_list.txt -c copy -hide_banner -y output.mp4

In Avidemux, you need a script to batch load the files. The number of videos files which can be loaded at a time is limited by physical memory, so loading 1460 files in a single step would likely not work. But Avidemux uses its own demuxers, so this solution could be considered as an alternative if the ffmpeg solution doesn't provide good results.

Don't expect VLC to be a good at remuxing or transcoding, it's not a video editor.

[ElusiveProcessing]

It should be possible with a ffmpeg concat file list.

from https://avidemux.org/smif/index.php?msg=90928:
(for %%G in (file*.ts) do @echo file '%%G') >> journey_list.txt
ffmpeg -f concat -safe 0 -i journey_list.txt -c copy -hide_banner -y output.mp4

In Avidemux, you need a script to batch load the files. The number of videos files which can be loaded at a time is limited by physical memory, so loading 1460 files in a single step would likely not work. But Avidemux uses its own demuxers, so this solution could be considered as an alternative if the ffmpeg solution doesn't provide good results.

Don't expect VLC to be a good at remuxing or transcoding, it's not a video editor.

Thank you! The ffmpeg method worked about 99% perfectly.

Here's the exact concat script that I used:

(for %%G in (*.ts) do @echo file '%%G') >> ConcatDB.txt
ffmpeg -f concat -safe 0 -i ConcatDB.txt -c copy -hide_banner -y output.ts
pause

Everything was joined in the proper order no problem, and video sync is perfect. It's the audio that's the problem. The audio is synced in the timeline fine, but it's just that every number of seconds (usually 10 seconds or less apart) I'll get what sounds like a very brief half second audio cut, as if the audio was muted for that very short time.

Could it be an issue with that I need to convert the .ts files to .mp4 before doing a concat? If that's the case, what would be the best program to get this done without any loss in quality?

[butterw]

Converting .ts to .mp4 is a lossless operation with ffmpeg or avidemux.

You should first double check whether the audio issue exists in the source files.

You could try loading a small number of files in Avidemux to check whether the issue is also present.

[ElusiveProcessing]

Converting .ts to .mp4 is a lossless operation with ffmpeg or avidemux.

You should first double check whether the audio issue exists in the source files.

You could try loading a small number of files in Avidemux to check whether the issue is also present.

So it's definitely the source files I think. Not sure what, but something. I put one segment into Avidemux and it played fine with audio, but then I appended the next segment and then the the whole combined clip had no audio including the first one that did have audio before I appended the second one. Then I cleared everything and opened only that second one and it still doesn't have audio even on its own. Yet it has audio if I play it with VLC.

That's something I'm noticing now is that almost every other segment has no audio on its own and also disables the audio on the initially opened segment when you append it onto the project in Avidemux, which I can only assume does indicate errors in the source.

There's not much I can do about the source unfortunately.

The final output file is watchable because like I said it's only half second audio cuts, but it is noticeable enough to the human ear where it can easily become distracting after only a little ways into watching. Not sure if there's some sort of audio smoothing trick out there, but otherwise I'm at the end here I believe. Thank you so much though for the help to this point.

[butterw]

Post the mediainfo properties of one of the .ts file.

Maybe re-muxing or re-encoding the source audio can solve the problem.

[jagabo]

Better yet, post three consecutive ts files that show the problem. Someone may figure out the cause and come up with a fix.

[blimey]

If these 1460 .ts files are from a streaming source, the most straightforward way to concatenate them is to use the corresponding playlist (m3u/m3u8).

Even if the playlist has (or lacks) url references, it still can be used locally after some simple editing.

[ElusiveProcessing]

Here are three consecutive .ts files. I also put a concat output of just those three files. The audio cut here is at the point where she's talking about her flowers being ruined.

[jagabo]

What player are you using? VLC plays output.ts with a little glitch. It plays much better in MPCHC. Try this simple append of the three files, made with

copy /b 1052.ts+1053.ts+1054.ts simple_append.ts

[jagabo]

Try this in a batch file:

del output.ts
fsutil file createnew output.ts 0
for %%G in (*.ts) do ( copy /b output.ts+"%%~nxG"=output.ts )

[blimey]

For simple binary concatenation, I got the same result as attached to post #10 (simple_append.ts)

Because the video track has variable frame rate, some players may have trouble, or assume the maximum frame rate.

Just remuxing the concatenated ts file to mp4 should eliminate this problem on most players.

[ElusiveProcessing]

Yeah I've actually only been using VLC. Didn't think to check how another player would do. So anyways that simple concat did the trick, and a remux to mp4 later and I now have the source fully reconstructed and with no audio cuts.

Thank you all very much for the huge help on this.

[jagabo]

Even the output.ts file from the ffmpeg concat batch file in post #3 plays OK in VLC if you remux it to MP4 or MKV.

[butterw]

Even the output.ts file from the ffmpeg concat batch file in post #3 plays OK in VLC if you remux it to MP4 or MKV.

Output.ts played fine in mpc-be and mpv. The issue was only really noticeable in vlc.

When one player has issues with a file but others don't, remuxing is typically the first thing to try (with ffmpeg, mkvmerge/mkvtoolnix, avidemux, etc.), or just use a player which plays it ok.

[ElusiveProcessing]

Ahh my bad, you're right. Just so I could see, I did that first script myself again but also did a remux to mp4 this time and it is in fact totally fine afterwards. And it was just VLC bringing out the issue in the initial ts output.