Steps & Settings for use of Whisper in Subtitle Edit ?

[Seeker47]

I just basically stumbled my way through the menus until I found a basic work process for this that seems to have proven successful for me. -- at least so far. But for anyone else who may be interested in more formal, detailed guides, I did find this online:

https://www.notta.ai/en/blog/how-to-use-whisper

[Subtitles]

I just basically stumbled my way through the menus until I found a basic work process for this that seems to have proven successful for me. -- at least so far. But for anyone else who may be interested in more formal, detailed guides, I did find this online:

https://www.notta.ai/en/blog/how-to-use-whisper

It is a good tutorial. Thanks for sharing.
I have been using Whisper AI for about a year and a GPU speeds things up a lot.
If you want to try Whisper AI with GPU anyway, and you have a Google Drive working, try using Google Colaboratory (Jupyter). You will need to install Whisper AI again but you have done it already on your PC so things will be easy for you. You will be able to use free GPU up to a certain size. Try it to transcribe a short video starting with the smallest models and move upwards till medium or large.
I haven't used it because my Google Drive is corrupted. Let us know if it works for you.
You might find this video helpful
https://www.youtube.com/watch?v=wrSelk44_Js&ab_channel=MathsChelsea

[Seeker47]

I just basically stumbled my way through the menus until I found a basic work process for this that seems to have proven successful for me. -- at least so far. But for anyone else who may be interested in more formal, detailed guides, I did find this online:

https://www.notta.ai/en/blog/how-to-use-whisper

It is a good tutorial. Thanks for sharing.
I have been using Whisper AI for about a year and a GPU speeds things up a lot.
If you want to try Whisper AI with GPU anyway, and you have a Google Drive working, try using Google Colaboratory (Jupyter). You will need to install Whisper AI again but you have done it already on your PC so things will be easy for you. You will be able to use free GPU up to a certain size. Try it to transcribe a short video starting with the smallest models and move upwards till medium or large.
I haven't used it because my Google Drive is corrupted. Let us know if it works for you.
You might find this video helpful
https://www.youtube.com/watch?v=wrSelk44_Js&ab_channel=MathsChelsea

Thanks for the suggestion. I've ordered a video card that has 4 GB of VRAM (probably not enough . . . ?), which might fit and work in this computer setup. So I expect to be giving that a try. I've only seen mentions of Google Drive, no exposure to that at all.

Right now, on CPU only, I have a Whisper job that's been running for 10 hours -- so far -- yet the Whisper log has only reached about 400 bytes in size. Should I take that as an indication that Whisper has stalled and given up ? Does the log only get written to at the very end ? Any previous job here of about the same size was completed overnight.

[Subtitles]

Are you using Subtitle Edit for this job or the Whisper AI command line?
Which model have you selected to do the transcription?
You can try the simplest model and see how it goes and go up higher once it finishes.
model medium and large are not going to work on your PC. You will see the difference when you install the GPU.

[Seeker47]

Are you using Subtitle Edit for this job or the Whisper AI command line?
Which model have you selected to do the transcription?
You can try the simplest model and see how it goes and go up higher once it finishes.
model medium and large are not going to work on your PC. You will see the difference when you install the GPU.

It just seems to be spinning its wheels on this job. Nothing has shown up yet in the upper left processing window, which I don't think was the case for the previous jobs.

Doing this under Subtitle Edit, which then hands off to Whisper. Using the Large model, as was the case for the previous 6 jobs -- all approx. the same size, and either with original French or German language. (No, wait -- there was one that was Japanese.) All of those succeeded, taking around 8 hours ea. to complete. Results ranged from satisfactory to quite good. I wasn't seeing any clear reason to deviate from this template that had worked several times, the only variable being the spoken language. But all of those had pretty clean soundtracks, with good recording and nothing much to interfere with that. I had not previewed the sound for this one so I don't know, but will go back to see where it stands.

[Seeker47]

After 12 hours and zilch to show for it, I pulled the plug on that job. May try it again after that video card arrives in a few days. Based on a quick check, I noted no obvious defects in the audio -- no people talking over each other, or poor recording, or obscuring background sound. So that was officially a first failure, once I'd worked out the rudiments of getting Whisper AI going.

[Subtitles]

After 12 hours and zilch to show for it, I pulled the plug on that job. May try it again after that video card arrives in a few days. Based on a quick check, I noted no obvious defects in the audio -- no people talking over each other, or poor recording, or obscuring background sound. So that was officially a first failure, once I'd worked out the rudiments of getting Whisper AI going.

Start again and use model small. At least you will get somthing even if it is not very accurate.

[Seeker47]

After 12 hours and zilch to show for it, I pulled the plug on that job. May try it again after that video card arrives in a few days. Based on a quick check, I noted no obvious defects in the audio -- no people talking over each other, or poor recording, or obscuring background sound. So that was officially a first failure, once I'd worked out the rudiments of getting Whisper AI going.

Start again and use model small. At least you will get somthing even if it is not very accurate.

I'll probably defer on this until that video card comes in -- enough other things going on in the interim anyway.

For the sake of comparison, has anyone experimented with Deepl ? If so, was it any good for the translating ? I had their Win standalone app installed, but only went to check it out for the first time a few days ago. At first the app could not be found (it had installed itself far under Users in C:, normally a place I would never install anything), and then when I tried to run it from that location it promptly uninstalled itself. Always possible that I made some mistake . . . . Seemingly a dead end, nonetheless.

[Seeker47]

@Subtitles,

Have you seen that bar graph which shows all the languages, and relatively how well Whisper performed in translating them ?
If not I'm sure I can find the link and post it here.

[Subtitles]

@Subtitles,

Have you seen that bar graph which shows all the languages, and relatively how well Whisper performed in translating them ?
If not I'm sure I can find the link and post it here.

Link
https://github.com/openai/whisper

I prefer to use the term trascription and not translation simply because I can check the final job while listening even in different languages.
For translation there are several options, including running Whisper again.

[Seeker47]

Well, I tried again, this time on an otherwise identical computer, but which has double the system memory (32 GB. -- and that maxes it out -- vs. 16), plus an MSI Video card that provides the GPU, along with 4 GB. of VRAM. And I'm baffled at how I'm now hitting a brick wall of greatly inferior results. The error messages make no sense to me at all. So I hope they might be revealing to someone else here.

-------------------------------------------------------------------------------------------------------------------

Job Failure 1:

SE: 4.0.0.0 - Microsoft Windows NT 10.0.19045.0 - 64-bit
Message: Calling whisper (Purfview's Faster-Whisper) with : C:\Portable SW\Subtitle Edit 4\Whisper\Purfview-Whisper-Faster\whisper-faster.exe --language fr --model "large-v2" --task translate "C:\Users\User1\AppData\Local\Temp\1708af13-1edd-4ff4-8d8e-b003f2a69c6a.wav"
Standalone Faster-Whisper r149.1 running on: CUDA
RuntimeError: CUDA failed with error out of memory

File "faster_whisper\transcribe.py", line 129, in __init__
File "D:\whisper-fast\__main__.py", line 537, in cli
File "D:\whisper-fast\__main__.py", line 655, in <module>
Traceback (most recent call last):
[9844] Failed to execute script '__main__' due to unhandled exception!
Calling whisper Purfview's Faster-Whisper done in 00:00:21.8710278
Loading result from STDOUT

--------------------------------------------------------------------------------------------------------------------

Job Failure 2:

SE: 4.0.0.0 - Microsoft Windows NT 10.0.19045.0 - 64-bit
Message: Calling whisper (Purfview's Faster-Whisper) with : C:\Portable SW\Subtitle Edit 4\Whisper\Purfview-Whisper-Faster\whisper-faster.exe --language de --model "large-v2" --task translate "C:\Users\User1\AppData\Local\Temp\85f936ba-c6b6-4964-b054-7165bb3b7841.wav"
Standalone Faster-Whisper r149.1 running on: CUDA
RuntimeError: CUDA failed with error out of memory

File "faster_whisper\transcribe.py", line 129, in __init__
File "D:\whisper-fast\__main__.py", line 537, in cli
File "D:\whisper-fast\__main__.py", line 655, in <module>
Traceback (most recent call last):
[4812] Failed to execute script '__main__' due to unhandled exception!
Calling whisper Purfview's Faster-Whisper done in 00:00:21.3626655
Loading result from STDOUT

---------------------------------------------------------------------------------------------------------------------

WHAT does this mean ?? I thought that the CUDA must be a native feature of most video cards like this. (?)

How the hell could I be running out of memory, when I never did on the original (lesser) rig -- that was minus these supposed advantages ?

[The above jobs were attempted using my existing "template" of Faster Whisper + Large Model.]

[Seeker47]

Another situation, and a different question. I've got at least a couple videos where there seem to be hard (burned in) EN subs, but some "genius" had the bright idea of placing them at the upper left of screen, and in about a 3 point font size, rendering them pretty much unreadable. What would be the best approach to Extract / Transform them to a readable size font / Replace them in the video ? (Preferably at the bottom, where they belong.) Can SE even extract and save hard subs . . . or perhaps using some other tool ? Failing that, they are easy enough to ignore as is, so generating an external .SRT file of them might be just as good. But I'm wondering if such a solution would just circle back to Whisper ?

[Subtitles]

Another situation, and a different question. I've got at least a couple videos where there seem to be hard (burned in) EN subs, but some "genius" had the bright idea of placing them at the upper left of screen, and in about a 3 point font size, rendering them pretty much unreadable. What would be the best approach to Extract / Transform them to a readable size font / Replace them in the video ? (Preferably at the bottom, where they belong.) Can SE even extract and save hard subs . . . or perhaps using some other tool ? Failing that, they are easy enough to ignore as is, so generating an external .SRT file of them might be just as good. But I'm wondering if such a solution would just circle back to Whisper ?

What is the video time duration?
What is the video quality resolution?
Hard subs are not easy to extract but there is a way if you are willing to spend few hours.
SE can't extract hard subs
Your best solution is to use Whisper to generate the subtitles.

[Subtitles]

Well, I tried again, this time on an otherwise identical computer, but which has double the system memory (32 GB. -- and that maxes it out -- vs. 16), plus an MSI Video card that provides the GPU, along with 4 GB. of VRAM. And I'm baffled at how I'm now hitting a brick wall of greatly inferior results. The error messages make no sense to me at all. So I hope they might be revealing to someone else here.

-------------------------------------------------------------------------------------------------------------------

Job Failure 1:

SE: 4.0.0.0 - Microsoft Windows NT 10.0.19045.0 - 64-bit
Message: Calling whisper (Purfview's Faster-Whisper) with : C:\Portable SW\Subtitle Edit 4\Whisper\Purfview-Whisper-Faster\whisper-faster.exe --language fr --model "large-v2" --task translate "C:\Users\User1\AppData\Local\Temp\1708af13-1edd-4ff4-8d8e-b003f2a69c6a.wav"
Standalone Faster-Whisper r149.1 running on: CUDA
RuntimeError: CUDA failed with error out of memory

File "faster_whisper\transcribe.py", line 129, in __init__
File "D:\whisper-fast\__main__.py", line 537, in cli
File "D:\whisper-fast\__main__.py", line 655, in <module>
Traceback (most recent call last):
[9844] Failed to execute script '__main__' due to unhandled exception!
Calling whisper Purfview's Faster-Whisper done in 00:00:21.8710278
Loading result from STDOUT

--------------------------------------------------------------------------------------------------------------------

Job Failure 2:

SE: 4.0.0.0 - Microsoft Windows NT 10.0.19045.0 - 64-bit
Message: Calling whisper (Purfview's Faster-Whisper) with : C:\Portable SW\Subtitle Edit 4\Whisper\Purfview-Whisper-Faster\whisper-faster.exe --language de --model "large-v2" --task translate "C:\Users\User1\AppData\Local\Temp\85f936ba-c6b6-4964-b054-7165bb3b7841.wav"
Standalone Faster-Whisper r149.1 running on: CUDA
RuntimeError: CUDA failed with error out of memory

File "faster_whisper\transcribe.py", line 129, in __init__
File "D:\whisper-fast\__main__.py", line 537, in cli
File "D:\whisper-fast\__main__.py", line 655, in <module>
Traceback (most recent call last):
[4812] Failed to execute script '__main__' due to unhandled exception!
Calling whisper Purfview's Faster-Whisper done in 00:00:21.3626655
Loading result from STDOUT

---------------------------------------------------------------------------------------------------------------------

WHAT does this mean ?? I thought that the CUDA must be a native feature of most video cards like this. (?)

How the hell could I be running out of memory, when I never did on the original (lesser) rig -- that was minus these supposed advantages ?

[The above jobs were attempted using my existing "template" of Faster Whisper + Large Model.]

Try model medium. model large is probably asking too much of the GPU

[Seeker47]

Another situation, and a different question. I've got at least a couple videos where there seem to be hard (burned in) EN subs, but some "genius" had the bright idea of placing them at the upper left of screen, and in about a 3 point font size, rendering them pretty much unreadable. What would be the best approach to Extract / Transform them to a readable size font / Replace them in the video ? (Preferably at the bottom, where they belong.) Can SE even extract and save hard subs . . . or perhaps using some other tool ? Failing that, they are easy enough to ignore as is, so generating an external .SRT file of them might be just as good. But I'm wondering if such a solution would just circle back to Whisper ?

What is the video time duration?
What is the video quality resolution?
Hard subs are not easy to extract but there is a way if you are willing to spend few hours.
SE can't extract hard subs
Your best solution is to use Whisper to generate the subtitles.

Now this is really strange, but interesting: I repeated Job #1 on the original, much less well-equipped computer, that has no GPU. There was no visible result for a long time, so I almost aborted this, thinking that it was never going to complete --- which has happened on a couple other jobs before. (Evidently the Whisper log does not get written until the very end, which would clarify one thing I was wondering about ? OTOH, on a couple of past jobs I could have sworn that subs text was gradually appearing in the SE box at upper left, so I can't be sure about that. Anyway, I decided to let this job run overnight. The next morning, lo and behold there was what looks to be a full .SRT file ! No idea how this managed to work on the "lesser" rig, but not on the maxxed out one that has an i7 3770 CPU and the MSI video card with 4 GB. of dedicated VRAM, vs. the "original" system that has an i5 3550 and no GPU ! The former system with the lesser CPU did not get overwhelmed !

For the last two jobs I had described above, having the same failures, those efforts bombed out within the first 30 seconds. My SE / Whisper settings have thus far remained constant.

Video Job #1 Duration: 57 minutes.
Video Resolution, per MediaInfo: 516 x 478, Matroska, AVC (High @ L4.1, CABAC)

I am certainly willing to experiment with a Medium Model instead of the Large . . . if that could actually be the variable that matters here.

Are there any specific CUDA settings ones needs to have Subtitle Edit set for, in this Whisper scenario ?

On another of those cases that has the miniscule EN hardsubs (and in the wrong screen location), I think I will be wanting to extract them.

[Seeker47]

Interested in any comments about or user experiences with the new options under Subtitle Edit, specifically the Large V3 language model and the added alternatives to Purfview's Faster Whisper. (Which has itself been updated a couple times fairly recently.)

[Seeker47]

I think Whisper can only handle one language per processing. (?) If there is a second language involved, is it possible to run a second processing pass just for that, and then to integrate those lines into a single SRT file -- with proper time sync for everything, of course ?

[Seeker47]

I'm also starting to experiment with the Sub Station Alpha format, instead. Google had suggested that that was a better way to achieve Yellow soft subs . . . which can be more desirable in cases where the standard White subs are tending to disappear against snow or other very light backgrounds in the video. According to Google, a lot of playback devices would support that format.

[kolloid]

Hey, stumbled on this thread because I had the exact same frustrations with Whisper in Subtitle Edit — the CUDA
errors, model confusion, the whole setup headache.

I ended up building my own transcoding tool (VidControl) that integrates Faster-Whisper-XXL directly. No Python setup,
no manual model downloads — you just pick a model size, hit go, and it handles the rest. VAD is on by default so
timestamps are usually pretty accurate out of the box.

@Seeker47 — regarding the yellow ASS subs: VidControl has a built-in ASS style editor. You can set color, outline,
font, alignment etc. and either keep them as soft subs or burn them in. Works well for the visibility issue you
mentioned with light backgrounds.

It's free if anyone wants to give it a try: https://vidctrl.de

[Seeker47]

Hey, stumbled on this thread because I had the exact same frustrations with Whisper in Subtitle Edit — the CUDA
errors, model confusion, the whole setup headache.

I ended up building my own transcoding tool (VidControl) that integrates Faster-Whisper-XXL directly. No Python setup,
no manual model downloads — you just pick a model size, hit go, and it handles the rest. VAD is on by default so
timestamps are usually pretty accurate out of the box.

@Seeker47 — regarding the yellow ASS subs: VidControl has a built-in ASS style editor. You can set color, outline,
font, alignment etc. and either keep them as soft subs or burn them in. Works well for the visibility issue you
mentioned with light backgrounds.

It's free if anyone wants to give it a try: https://vidctrl.de

Thanks. I'm game to try other possible and previously unseen solutions. Someone here was advocating Tero Subtitler, if I got that name right. I downloaded it and installed it, but haven't made any headway with it as yet. I've long had Subtitle Workshop, even after another group took over the project, and have slowly been getting more comfortable with some aspects of what Subtitle Edit has to offer.

I just posted the following in another forum, as part of a reply to a fellow member there:

"There are different roads one might take to a destination.
Query: Suppose we could make purple subs in a bold font, in let's say a 16 point size, and have that display as intended during playback. Could that obscure the existing hard subs, even without a preparatory masking bar step ? I'm thinking this would be fugly, but would it also obscure anything else in the frame that we did really want to see, or otherwise be too distracting ? "

Guess that would depend on the particular film / video. (I may actually have done something like this once in experimentation, totally by accident and without knowing how.)

Additionally, so far, it is looking like the SSA subs are just being ignored by the Oppo BR deck upon playback. That is my "Go To" option in the video chain, except for HEVC / x65 sources, which it does not handle. But there are a couple other options available here for me to investigate, including the Nvidia Shield.

[Seeker47]

@kolloid,

The info on your site certainly appears intriguing. I guess my concerns will revolve around a couple things: complexity / learning curve, and then the transcoding -- which I suppose is going to be unavoidable. A number of tools found on VH will stress "No Transcoding Required." Even among some video editors. The implication being that the transcoding invariably introduces some degradation in visual quality. (?)

[kolloid]

@Seeker47 Fair concerns!

For what you're trying to do — adding styled ASS soft subs to a video — you actually don't need to transcode at all.
VidControl has a "Copy" mode that just re-muxes: the video and audio streams stay untouched (zero quality loss), it
only adds or replaces the subtitle track. Takes seconds instead of hours.

You'd only need to transcode if you want to burn the subs into the video (hardcoding). But for soft subs, copy mode is
the way to go.

As for complexity — the UI starts with the basics visible and tucks the advanced stuff into collapsible sections. For
your use case it's basically: drop file in, set subtitle style, pick copy mode, go.

Regarding your Oppo ignoring SSA subs — that's usually a player/container compatibility issue. MKV + ASS soft subs
work on most software players (VLC, mpv, Plex etc.) but standalone hardware players are hit or miss with ASS. If your
Oppo doesn't support it, burning them in would be the fallback — and yeah, that does require a transcode, but at CRF
18-20 you won't notice a difference visually.

[Seeker47]

Thanks kolloid. I can play around with this, but I know of a couple other likely-to-be-interested users who would benefit from a tutorial, be it a video or PDF guide. I'm at least on the borderline of that status myself. I could not even see where to close the program, but then remembered the old, universal ALT-F4 (which might go back as far as Windows 3.0, or even earlier ?) which worked.

[EDIT: Ah, I see that you have a page here on VH in the Software section, and it does mention a video tutorial. But I went to that YT link, and did not notice any that seemed relevant to this program.]

The program seems to be the equivalent of a portable version, in that it looks like one can locate it anywhere -- in my case, that was on a data drive, because the boot SSD is always getting too close to the "red zone" for fullness, and I keep putting off migrating the OS to a larger one -- and your program does not require installation, or Registry entries ? (I'm a major fan of portable versions, which one can take along with them on a flashdrive and run them from it.)

[EDIT: I also see that the program is formally described there as being Portable.]

[kolloid]

Hey Seeker47, thanks for trying it out!

Close button: That was a known issue — it's been fixed in v2.7.2. The title bar and close button should now be clearly
visible.

Tutorial: Fair point, I'll put together a short video walkthrough covering the basics.

Portable: Exactly, fully portable. No installation, no registry, no leftover files. Runs fine from a USB drive too.

Thanks for the feedback!

[Seeker47]

So far, a mixed report card on the SSA Yellow subs experiment. I've done that for a feature length documentary, for which this did work upon playback with one of the Dune HD models. Also a dramatic feature film and a short, for which it did NOT work, either on the Oppo BR deck, or on the Dune. (Unless I messed up somehow, in making those subs . . . . ) Haven't tried to play these yet with either the ancient WD Live TV player box, or on the Nvidia Shield. Chances are they should work on the latter, since it is running VLC, which is known to support this.

And I'll be intending to give this a shot soon using the VidControl program for making the subs, instead.

[Seeker47]

Hey Seeker47, thanks for trying it out!

Close button: That was a known issue — it's been fixed in v2.7.2. The title bar and close button should now be clearly
visible.

Tutorial: Fair point, I'll put together a short video walkthrough covering the basics.

Portable: Exactly, fully portable. No installation, no registry, no leftover files. Runs fine from a USB drive too.

Thanks for the feedback!

Any progress on that tutorial ? As I may have mentioned, I happen to favor the step-by-step PDF approach. The video ones tend to zoom by too quickly, so you have to keep stopping to rewind. The PDF ones users can follow at their own pace.

If this can also get into the masking-out of hardsubs in preparation for the new SRT ones (?), so much the better ! I'd be glad to make suggestions re improvements for any such tutorial you might come up with.

[kolloid]

Hey Seeker47, great feedback!

For a Tutorial: You're right, a PDF guide makes more sense - easier to follow at your own pace. I'm working on a
step-by-step PDF covering the main workflows (file loading, track configuration, encoding, OCR/subtitle workflow).
Should be ready soon.

For Hardsub masking: That's actually a great idea and something I've been thinking about. I'll look into adding a subtitle
region mask feature - the idea would be to let you select the hardsub area and have it masked out during encoding.
Stay tuned!

Happy to hear any other suggestions in the meantime.

[Seeker47]

Thanks kolloid. Good news, and I know a couple others who will be glad to hear this.