Muxing of two slightly different video sources

[KennyTNT]

Hi, I've actually been thinking if this post should have gone to the video or the audio editing forum or if i should have done two different and post it to both.
Basically, I have two rips of the same tv show. One is from the official blue-ray, however it doesn't contain my language, because copyright holders in my country decided they wouldn't release a blue-ray or a dvd copy. The second one is a television recording in my language.

What is the problem?

In the original there are three to four scene changes with no sound and a black screen lasting about a second, this have all been removed in the television broadcast.
The television broadcast also adds a commercial at about half the episode.

What I basically would like to do is automagically change the tv audio track to be on par with the blue-ray video or automagically cut the video track to be on par with the tv audio.

This are some of my thoughts on how this could be handled, since i have to do this on hundreds of episodes, each episode requiring me about 20 minutes of editing with Audacity, I would like for this to be as automated as possible.

1) Re-encode the blue-ray video with FFmpeg and a filter to detect and remove the one second black transitioning. This could work IF the television broadcast didn't have a commercial. Additionally, using also software to cut commercials from the tv broadcast could result in frame differences. The disadvantages of this solution would be the re-encoding of the video, the false positives (like scenes which intentionally include black frames) and the fact that the automatic removal of the commercial from the tv source could delete actual content frames and mess up the sync

2) Split both video sources at every sudden scene change. While at first this sounded like a great idea I tried it with ffmpeg but however using the same value for the scene change provided very different results. While in blue-ray copy the sudden scene changes are well done and well defined, in the tv rip a sudden transition appears as if the two frames where overlapping. I think I may give PySceneDetect a go since it's seem to offer different algorithms to detect scene changes but I'm not too sure that this it's going to work

3) Extract every single frame from the blue-ray and the tv. Select the first frame they share in common, than a picture comparison tool, like imagemagick could detect similarities between frames. Let's say the first matching frames are x and y. The script would match x and y, x+1 and y+1, ..., x+n and y+n. Once they are out of sync, find the next identical sequence. Many things might go wrong with this approach. The hardest thing would be to find a matching algorithm: since the videos are from two different sources, the aspect ratio is different (16:9 vs 4:3) and also the quality is very different. I took a look at some algorithms for the task, for example a keypoint extraction tool for matlab. This would have the advantage of not relying on colors aspect ratio, but I'm not really sure on how could I compare the two

4) The last approach I came up with was a sound only approach. There could be a tool that recognizes similar particular sounds between two audio files like a siren or another distinct sound and helps sync the two. I downloaded the pluraleyes trial, since they advertised to be the very best but I was utterly disappointed. It only seemed to work on almost identical audio tracks. Since that wasn't the case for me (a dub in a different language) it doesn't even try

Is there a software that can help me out, or i'll have to stick to the good 'ol Audacity?

[poisondeathray]

What does it have to be "automatic" ? There is lots of room for error given your description of the problem

For this type of task I think it would actually be easier to do manually in a video editor. 4 scene changes , a 1 sec black screen, a commercial is not a lot of work do. It would probably take longer fiddling with your threshold settings with some detection filter, and that would still likely mess up some frames.

I'm assuming the blu-ray video is higher quality . If it were me, I would want to keep the blu-ray video and add the additional audio as another track . Since that audio is shorter, you just have to add silence / mix it with the scene changes, cut out the commercial audio, and line it all up . At least with a video editor you can check for accuracy and make adjustments . You can visually check and hear. There is feedback immediately. For example, if some frames are dropped in the broadcast you will easily see where and when and can make adjustments. With some "automatic" method, likely it's going to be very finicky because it won't be perfectly accurate. Lots of going back and forth making adjustments.