How to automatically translate the audio-track of a video?

[cronos111]

Hello, I'm a Spanish speaker.

There's a lot of high-quality content online in English, but unfortunately, I don't understand it.

I'd like to use an audio processing program with automatic translation, similar to the one on YouTube.

That is, I have an MKV video with the audio track in English, and I want a program that will translate the audio track into Castilian Spanish (not Latin American Spanish).

Then I use MKVToolNix to add the new audio track to the MKV.

And that way, I get a multi-audio video (english + spanish)

The only way I can think of to do this would be to upload the video to YouTube, get the audio track automatically, and then download it. But I'd like to avoid doing that; I prefer to do it locally.

Is there a program that can do this?
I know how to extract the audio track and remultiplex it.
I just need to generate the new, translated audio file.

thank you so much

[kolloid]

You can use my tool, VidControl. You just need ollama or lmstudio for local translation.

[cholla]

@ cronos111,
I do not know if these are what you need.
Do they even seem to be ?
https://www.transyncai.com/blog/voice-in-spanish-translation-2026/

https://www.adobe.com/products/firefly/features/translate-audio/english-spanish.html

@ kolloid,
I have not used VidControl.
It is probably a good program but I'm certain it does not do what cronos111 wants.
Unless I'm wrong he wants to input an English audio file & get an output Spanish audio file. Preferably a Castilian Spanish audio file.

[davexnet]

This is a can of worms. Better off using subtitles

[cholla]

This is a can of worms. Better off using subtitles

I sure can't disagree with you dave.
I have heard of a close to universal translator for cell phones.
I have not tried to check it out.
I do not believe technology is there yet.
A little too Star Trek.

[kolloid]

Oh, yeah i totally misread the topic. My bad

[GAhere]

I suspect, one would have to use an audio (speech) to text translation program, the result would give you a text file.

Then you could use a text to speech (audio) converter/reader to create a audio file in your native language.

Then somehow you would need to combine the audio file into the video stream via a video editor and hope that you managed to sync everything so the converted audio is in time with the scenes..

The converters are available, but man, what a lot of work and the result is going to look funny when folks are talking and the mouth/lips movement is not in sync by even a couple of milliseconds..

I would have to agree, Subtitles would be a better choice..

[cholla]

I had thought of the method in GAhere's post.
I agree with the reasons posted but wanted to add another:
All the voices would be the same. Not even a difference between a man or woman.

[cronos111]

Thanks for the replies.

I tested Transync AI v2.1.1 (free version). I was pleasantly surprised. The quality of the English-to-Spanish translation is much better than the automatic translator that comes with Google.

What I did was play a YouTube video in Firefox. The audio source selected in Transync AI was the computer's audio.

The program generates translated text from the system audio. The translation is quite good, and timestamps are included. Unfortunately, I didn't find any options to export the text as an audio file within Transync AI.

So, I have three main problems with Transync AI:
1) I can't save the translated text in AAC-LC format, for example. I suppose I'll need some other program to do this: read the text and save it as AAC.

2) The speed at which the translated text is read needs to be variable to synchronize correctly with the original video. I suppose I'll need some other auxiliary program to adjust the playback speed and synchronize it with the video.

3) All of the above complicates video editing. I'd simply like to add the new audio track using MKVtoolnix. My video editing skills are very basic.

Is there a version of Transync AI that allows me to do all of this at once? I don't mind paying for it; it's worth it to save me many hours of work.

In other words, I want this to translate the audio of online videos from English to Spanish. This would save me from having to read the subtitles, which is very annoying because then I can't watch the video.

Ideally, I'd like to select the MKV video directly from Transync AI, and then have it extract the English audio, translate it, and generate the new Spanish audio track. And if it also generates an SRT file with synchronized subtitles, that would be perfect.

I haven't tried Adobe Firefly yet; I don't know if it includes any of these options. I'll look into it too.
Basically, I'm looking for an English to Spanish dub.

Thank you so much!

[cronos111]

I forgot to mention an important detail.

The translation quality of Transync AI far surpasses the automatic subtitles integrated into YouTube.

What I'm trying to say is that Transync AI can be a very interesting option for generating high-quality subtitles to add to YouTube videos.

The only problem would be synchronizing those subtitles with the original video. But Transync AI generates timestamps, and that can be very helpful.

Personally, I don't know how to process the text file generated by Transync AI to convert it into a functional SRT file suitable for embedding in the video's MKV file.

If anyone knows how to do this, that would also be very interesting.

[cronos111]

I've been testing Adobe Firefly, and honestly, I haven't liked it at all; it's awful.

Here's why:
- It requires a paid account; there's no free version.
- There's a 10-minute limit on the original video.
- It doesn't accept MKV files; you have to convert it to MP4.
- It also doesn't seem to accept YouTube Opus audio.
- A 9:07 video plus English to Spanish translation uses 2,739 credits, which are deducted from your monthly balance.
- If you close the work tab before finishing, the process can freeze.
- If the job freezes, you have to repeat it, but the credits aren't recovered.
- It's not very compatible with Firefox; you have to use Chrome. It's very slow at both transcription and video generation. There are transcription errors from English to Spanish; sometimes it doesn't detect the source language.

IMPORTANT: Adobe analyzes the video content, and if it deems it harmful, it won't translate it!
"This content does not comply with our User Guidelines."
https://www.adobe.com/es/legal/licenses-terms/adobe-gen-ai-user-guidelines.html

I've tried with 10 videos on different topics downloaded from YouTube, and I always get the warning that "This content does not comply with our User Guidelines."

They weren't violent videos or copyrighted films; they were varied content: news, interviews, reviews, technical tutorials. I don't understand why Adobe is blocking everything. Not a single video has worked for me.

What content is supposed to be translatable?
Is everything downloaded from YouTube blocked?

[kolloid]

VidControl can generate subtitles and generate voice from subtitles.

[cronos111]

VidControl can generate subtitles and generate voice from subtitles.

Thanks for the suggestion. I just downloaded version 2.9.33 (June 30, 2026) to try it out.

Unfortunately, I get an error when trying to open the file. I get a message in German. I've attached a screenshot.
My videos play without problems in VLC; they don't appear to be corrupted. They were downloaded from YouTube using yt-dlp. I don't know if there's any incompatibility. I'm using an updated version of Windows 11 x64.

¿Is it possible to copy the text file generated in Transync AI into VidControl? for, audio-auto-srt, does VidControl only work with its own subtitle generator?

Thank you very much!


---------------------------
Files Unreadable
---------------------------
Some files could not be loaded:

• Build an empire [EN].mp4 (unreadable/corrupt)

Possible causes:

• File is damaged/corrupt
• Format is not supported
• File is encrypted or protected
---------------------------
Accept
---------------------------

[kolloid]

Hmm Interesting, do you have any further information to help me analyze the problem here? Or maybe the file itself?

For your question - i need to check it out and reply if i have further information.

[cholla]

This is another I found searching with Google.
Like Transync AI or Adobe Firefly: I have not tested this.

https://www.kapwing.com/subtitles/audio

[cronos111]

Hmm Interesting, do you have any further information to help me analyze the problem here? Or maybe the file itself?

For your question - i need to check it out and reply if i have further information.

Thanks for the help. I'll try it with another file for now and see if it works. If it fails again, I'll start a new thread so as not to mix topics in this one

This is another I found searching with Google.
Like Transync AI or Adobe Firefly: I have not tested this.

https://www.kapwing.com/subtitles/audio

Thanks for the help, I'll look into it carefully and if I have any questions I'll start a new thread so as not to mix topics.