LocalVocal: Local Live Captions & Translation On-the-Go

LocalVocal: Local Live Captions & Translation On-the-Go v0.2.6

Go to download
Overview Updates (13) Reviews (3) History Discussion
L

LittleGelato

New Member
royshilkrot said:
Thanks for trying it out!
I'm also very glad you got it working.
In general, I would be careful about running multiple instances of the filter especially GPU since it loads up cuda memory and that's buggy sometimes.

But let me just understand
You're speaking one language or two?
In general the non English models are multilingual and should pick up any language. I think there's an "auto" language where it tries to guess the spoken language.

Anyway you've asked about the captions and translation
The answer is yes that should be available - sending the original caps one way and the translation another way so you have a choice. That's coming up as a feature
Click to expand...
Sorry for the late reply!
Of course, it's a super useful tool and I'm excited to see how it evolves. That's good to know about the GPU memory issues, maybe I'll try to use the CPU versions and see if it gives better results when using the larger models and multiple at a time.
Does anyone know if there are conflicts if you try to use the CPU AND GPU versions at the same time?

To answer your question about my situation, I’m a native English speaker with 10 years of experience learning Portuguese (BR). I explore Brazilian content and culture on my stream, which means I’ll occasionally mix both languages. My goal is to provide closed captions (CC) translated in both English and Portuguese so that viewers who only speak 1, or people with hearing issues, can read them no matter the language I speak.

The feature that you mentioned that's up for review would be amazing! For the moment I think I can use a twitch overlay extension to auto-transcibe my stream into Eng and use LV to translate to PT to send to twitch's built-in CCs.

I'll be checking out your other plugins, keep up the great work!
 
royshilkrot

royshilkrot

Member
LittleGelato said:
Sorry for the late reply!
Of course, it's a super useful tool and I'm excited to see how it evolves. That's good to know about the GPU memory issues, maybe I'll try to use the CPU versions and see if it gives better results when using the larger models and multiple at a time.
Does anyone know if there are conflicts if you try to use the CPU AND GPU versions at the same time?

To answer your question about my situation, I’m a native English speaker with 10 years of experience learning Portuguese (BR). I explore Brazilian content and culture on my stream, which means I’ll occasionally mix both languages. My goal is to provide closed captions (CC) translated in both English and Portuguese so that viewers who only speak 1, or people with hearing issues, can read them no matter the language I speak.

The feature that you mentioned that's up for review would be amazing! For the moment I think I can use a twitch overlay extension to auto-transcibe my stream into Eng and use LV to translate to PT to send to twitch's built-in CCs.

I'll be checking out your other plugins, keep up the great work!
Click to expand...
Multiple open captions (ones that appear on yout stream, not "closed caption" that people enable in e.g yt or twitch) are possible with Text Sources, for example you send captions to one text source and translate to another text source.
 
L

LittleGelato

New Member
darkhelmet said:
is there any simple profanity filter, or would that have to all be entered in the exclusions list?
Click to expand...
It doesn't specifically mention the Whisper models but the link shows that OpenAI uses context to decide if a word should be censored or not. If it falls under one of the provided categories then it should block it, but if you just say a word without context it will not block it. I was realizing it wasn't consistent and stumbled on this info about how it filters.
 
royshilkrot

royshilkrot

Member
LittleGelato said:
It doesn't specifically mention the Whisper models but the link shows that OpenAI uses context to decide if a word should be censored or not. If it falls under one of the provided categories then it should block it, but if you just say a word without context it will not block it. I was realizing it wasn't consistent and stumbled on this info about how it filters.
Click to expand...
it could be done a lot simpler if you have a dictionary of expletives that you want to block and the plugin will just compare against it and replace with *****
 
K

Kentz

New Member
Hi,roy!I am a newbie in obsstudio. I want to use localvocal to realize the function of live broadcast and real-time translation on the streaming media platform to explain the product. I have installed the obsstudio software and the localvocal plug-in, but I do not know how to set localvocal in obsstudio to achieve real-time Translation function, so I came to consult you. If you see the email, reply to me. Thank you very much! God bless you
 
ora

ora

New Member
I have audio from multiple sources at once. Is it possible to have more than one source with the filter active at the same time and have them merge into a cohesive set of captions?
 
royshilkrot

royshilkrot

Member
ora said:
I have audio from multiple sources at once. Is it possible to have more than one source with the filter active at the same time and have them merge into a cohesive set of captions?
Click to expand...
It's possible yes but resource intensive so depends on your hardware
You could stitch the captions with "name: .." after the fact, but I don't currently have a built in solution for that

In the future I will add the speaker diarization so you could avoid all of this and have just one stream
 
royshilkrot

royshilkrot

Member
royshilkrot updated LocalVocal: Local Live Captions & Translation On-the-Go with a new update entry:

v0.2.6 - More robust, better translation, Linux fix

A big release. Fixing things and adding evaluation tools
  • Fixing Linux problems with missing dependency
  • Testing tools and scripts for evaluating the algorithm
  • Add Whisper-based translation
  • Add M2M100 1.2B model for translation
  • Optimizing overlap regions
  • Sentence suppression fixup
Download
Click to expand...

Read the rest of this update entry...
 
A

adamesek

New Member
Good job!!
I have been testing this solution since the beginning of LocalVocal. Using English - it works very well. No problems.
However, we do not need the use of English.
Unfortunately, in Polish, characters such as - ś ż ł etc. do not appear. This causes 'entire expressions to disappear'. I reported this on one of the first versions. .. and that's how it stayed.
MacBook M1 Pro
 
L

LittleGelato

New Member
Hey there, the new update is looking lighter on the system.
I believe I'm having issues with the new whisper translation though. It seems more accurate/faster but when I add the translate option and select a text source for the Translation Output, a different one from the Subtitles Output, it seems to send the translated text to the Subtitles Output anyways and sends the original transcription nowhere. I deleted both text sources and remade them to no effect. Any thoughts on what I might be missing or on what I could send to help out?
 
royshilkrot

royshilkrot

Member
adamesek said:
Good job!!
I have been testing this solution since the beginning of LocalVocal. Using English - it works very well. No problems.
However, we do not need the use of English.
Unfortunately, in Polish, characters such as - ś ż ł etc. do not appear. This causes 'entire expressions to disappear'. I reported this on one of the first versions. .. and that's how it stayed.
MacBook M1 Pro
Click to expand...
@adamesek strange - i thought i took care of those UTF8 problems! do you want to open another issue and provide some examples? you can also enable the logging level to be INFO and send over the log which provides a lot more information
 
royshilkrot

royshilkrot

Member
LittleGelato said:
Hey there, the new update is looking lighter on the system.
I believe I'm having issues with the new whisper translation though. It seems more accurate/faster but when I add the translate option and select a text source for the Translation Output, a different one from the Subtitles Output, it seems to send the translated text to the Subtitles Output anyways and sends the original transcription nowhere. I deleted both text sources and remade them to no effect. Any thoughts on what I might be missing or on what I could send to help out?
Click to expand...
@LittleGelato very strange. this is probably a bug. i'd like to see some more information so i can replicate and squash the bug. can you send some info on https://discord.gg/Asv3ry3v7q ? thanks
 
A

adamesek

New Member
royshilkrot said:
@adamesek strange - i thought i took care of those UTF8 problems! do you want to open another issue and provide some examples? you can also enable the logging level to be INFO and send over the log which provides a lot more information
Click to expand...
logs, attach
royshilkrot said:
@adamesek strange - i thought i took care of those UTF8 problems! do you want to open another issue and provide some examples? you can also enable the logging level to be INFO and send over the log which provides a lot more information
Click to expand...
logs, attach
 

Attachments

  • 2024-05-14 09-23-08.txt
    263.1 KB · Views: 2
You must log in or register to reply here.
Top