Auto convert Japanese speech into subtitles?

[CompVid]

Is there a way to automatically convert Japanese audio to English language subtitles? Either to do it 'on the fly' as the video is played or to process the video separately and so create a .srt file or similar?

[Subtitles]

Is there a way to automatically convert Japanese audio to English language subtitles? Either to do it 'on the fly' as the video is played or to process the video separately and so create a .srt file or similar?

You can use Whisper AI to translate Japanese audio to text and create either Japanese subtitles (Not recommended) or transcribe to Japanese and then translate using Subtitles Edit
https://github.com/openai/whisper
You will need a good computer with a GPU of 8GB VRAM if you want to get good accuracy.

[timtape]

Yes speech to text has been around for a while but it's not always appreciated that if humans struggle to understand what is being said, so probably also will the software. The software makes less mistakes when the captured speech is as clear as we can make it.

[CompVid]

Is there a way to automatically convert Japanese audio to English language subtitles? Either to do it 'on the fly' as the video is played or to process the video separately and so create a .srt file or similar?

You can use Whisper AI to translate Japanese audio to text and create either Japanese subtitles (Not recommended) or transcribe to Japanese and then translate using Subtitles Edit
https://github.com/openai/whisper
You will need a good computer with a GPU of 8GB VRAM if you want to get good accuracy.

Thanks

Hmm, my GPU only has 6GB of VRAM .......

Also, if I use Whisper AI to transcribe to Japanese to text first, how does it handle the timings regarding when the subs appear and for how long (when converting to subtitles)?

[Subtitles]

Is there a way to automatically convert Japanese audio to English language subtitles? Either to do it 'on the fly' as the video is played or to process the video separately and so create a .srt file or similar?

You can use Whisper AI to translate Japanese audio to text and create either Japanese subtitles (Not recommended) or transcribe to Japanese and then translate using Subtitles Edit
https://github.com/openai/whisper
You will need a good computer with a GPU of 8GB VRAM if you want to get good accuracy.

Thanks

Hmm, my GPU only has 6GB of VRAM .......

Also, if I use Whisper AI to transcribe to Japanese to text first, how does it handle the timings regarding when the subs appear and for how long (when converting to subtitles)?

You can try it with your current GPU but you will need to select the less accurate models
See the list shown in the link I gave you above. You can try model medium but I doubt if you would succeed. Model small should be OK but you will lose accuracy. The list mentions for model medium you would need 5GB VRAM which theoretically should be OK as you have 6GB VRAM but I think it will not work. Hope that you can prove me wrong.


Available models and languages

There are six model sizes, four with English-only versions, offering speed and accuracy tradeoffs. Below are the names of the available models and their approximate memory requirements and inference speed relative to the large model. The relative speeds below are measured by transcribing English speech on a A100, and the real-world speed may vary significantly depending on many factors including the language, the speaking speed, and the available hardware.

Size Parameters English-only model Multilingual model Required VRAM Relative speed
tiny 39 M tiny.en tiny ~1 GB ~10x
base 74 M base.en base ~1 GB ~7x
small 244 M small.en small ~2 GB ~4x
medium 769 M medium.en medium ~5 GB ~2x
large 1550 M N/A large ~10 GB 1x
turbo 809 M N/A turbo ~6 GB ~8x

[VoodooFX]

Is there a way to automatically convert Japanese audio to English language subtitles? Either to do it 'on the fly' as the video is played or to process the video separately and so create a .srt file or similar?

https://github.com/Purfview/whisper-standalone-win

[Asdfgf]

I have made many subtitles using automatic captions on capcut. For me is the best way to do that, but sometimes if the audio is not good, or the speaking is very fast, it can fail a bit.

[CompVid]

Thanks. I've never used Capcut - is it easy to set up to convert Japanese audio to English subtitles?

[syberdude]

I've used subtitle edit program to good results when translating subtitles or getting ai to work to transcribe them. I think the program uses whisper of some sort and mpv to make it all happen with success.

https://www.videohelp.com/software/Subtitle-Edit

potplayer can do it but it's far too slow in doing so I feel but it's an option.

There is also Terosubtitler but I haven't used this one myself. Generally all these programs will use whisper in some way for translation using ai and such.

[Asdfgf]

Thanks. I've never used Capcut - is it easy to set up to convert Japanese audio to English subtitles?

I have never tried for a japanese video, but i will make a try and upload the results

[Asdfgf]

And here are the results. I used this video as base: https://www.youtube.com/watch?v=JYEutL_zQQc

It is a 5 minutes japanese speaking video. It has its own subtitles on youtube. This is the original youtube subtitles file: https://pixeldrain.com/u/xkMbJxV3

This is the japanese capcuts auto-captions (japanese): https://pixeldrain.com/u/YhKgnYc5

And this is the auto-translation to english (using this web https://subtitlestranslator.com/en/ ): https://pixeldrain.com/u/ksFHRhmj

As you can see, is not perfect but enough to understand the video. If you are very interested on an specific video, maybe you can combine whisper app (i have never tried it, but other users recomended it), and this method with capcut to generate the subtitles timeline.

EDIT: I have seen whisper, and wow, that tool seems to ve very powerful. I have never used it, as I said, but will read about it.

[CompVid]

Excellent, thanks for the suggestions and help from you all.

BTW, just FYI, Malwarebytes is flagging that subtitlestranslator.com site as containing Malware. Have you had any issues with it?

[Asdfgf]

Excellent, thanks for the suggestions and help from you all.

BTW, just FYI, Malwarebytes is flagging that subtitlestranslator.com site as containing Malware. Have you had any issues with it?

I have been using that web for years and no problem.

[CompVid]

Okay, possibly Malwarebytes being twitchy or the site is on a 'list'.