Version 3.0 released! A free trial is available here! patrikzudel.me/ai-subtitles Changelogs: - Added support for the "Large-v3" model. - Added "ignoredSentences.txt" to the program folder. Adding sentences to this file forces the program to ignore them. Each sentence should be in a new line. Useful for removing unwanted "Thanks for watching." and similar.
Hi, Patrik! And thank you for such an amazing tool! I just have a question about the "thanks for watching" that appears even when I'm not talking. I already added it to the "ignored sentences" file, but it continues to show. Am I doing something wrong? Thanks in advance!
Trocha iné video, ako obvykle, ale potreboval som ho vydať! Je to o mojom programe, ktorý používam na streame na pridanie real-time prekladu do Angličtiny.
Version 2.0 released! A free trial is available here! patrikzudel.me/ai-subtitles Changelogs: - Added FP32 support (Should increase the compatibility for older NVIDIA GPUs) - Added ability to enable output the transcription to file. You can enable it in the settings. - Managed to greatly decrease the size of the program to under 1 GB!
Hi! I bought the software to support development. I'm guessing right not the translation only works from source language to English? I hope in the future there might be an option to translate from English to another language? This would be great. Good luck with your development!
Hi there, any way we can contact you to develop some custom stuff? we need to subtitle and translate english videos to spanish and whisper seems to be really good.. can we contact you somehow?
Would this work on a MS teams webinar or meeting ? Can it detect the input language without presetting ? Our presentations are hybrid Dutch-French, so when one language is spoken I would like to have subs from the other language and vise versa..
Hi, I'm having a small issue. If I delay the scene and the microphone, obviously delaying the scene also delays the subtitles as those are located in the same scene. So delaying doesn't really do anything as the purpose of delaying in the first place was to try to match up the speech and the subtitles to happen at the same time. How could I solve this problem and make the subtitles happen at the same time I'm speaking? Thanks in advance.
You need one more scene where you put the undelayed subtitles and delayed "main" scene (using "Scene" source). If you have any issues getting this to work I recommend messaging me on Discord (patrikzero) or Ko-fi. I'll gladly help ^^
Awesome! I just purchased. Wanted to ask, any way to get the output text file live? I’d like to have a text to talk for people who are being translated for but I’d like it to be read to them
You could write a short python script that would do that. In settings you can turn on live output to a file but it is only the last line. So you'd need a script that takes every new line and puts it into a file, ChatGPT should be able to do that very easily.
to use this do you only need the program? there was an extension for twitch like this but you had to use the google cloud thing and it was kind of expensive
This program looks awesome and I would gladly pay for it, I'm curious though, I have a linux NAS that runs dockerized containers, is there anyway it can be installed on my NAS and I can utilize that PC for the subtitles. It looked like you did something along those lines in your dual pc set up but just curious if specifically a dockerized linux pc could install/run it?
Hey ! Thank you very much for that plugin , just bought it today. As a french streamer but who's often speaking english to my mates i would love to be able to choose to translate english to french so that my french viewers could understand me as well. Am i dumb and i haven't found the option or is it not a thing ? If it's not are you planning on adding that to a new version ?
Hey I purchased your AI translator, but now when I open it up, it is no longer transalting for me. It is stuck on "Serving subtitles at 192.168.2.141:8081 Enter this into the Browser source in your OBS." and then hangs. Nothing is happening.
@@PatrikZero it is a laptop tbh, and i just changed display orientation for a minute to do the settings and save. and yess i wish for scrollin in the next version, thanks for replying and thanks for your hardworking ❤️
Hello, thanks for your video. I speak French, English, Chinese and Japanese in my stream. I was wondering if there's an option to switch input language quickly so that the software can pick up what I say and understand the right language. I also need to have several output languages at once for my stream, for instance when I speak French I'd like to have subtitles in English, Japanese and Chinese at the same time. I know your software may not be able to do that yet, but do you have suggestions of solutions maybe? Thanks anyway.
This has been my struggle. There is no ready made tool for this that I can find. Patrik is the only person I've seen that takes into account a fully bilingual setup. I want my stream to display Japanese subtitles when I am speaking English, and English subtitles when I am speaking Japanese. You'd think it was simple, and honestly it probably is once you already have the translation pipeline in order, but nobody seems to put in the option to have multiple instances simultaneously running translations, and I haven't seen an option to reject a specific detected language from subtitles. I don't want my broken japanese subbed with Romaji or my English showing up as gibberish Katakana. And developing your own solution without experience with Cuda, Tensor, and all the other services is.. really mind numbing. I'd be happy to pay somebody like Patrik who actually develops a robust application.
Hello! Do you have plans to make this translate from English to another language? I bought this because I wanted to translate my future streams from English to Portuguese, but I just saw that it's not possible for now 😭 that's on me for not reading ig
I've heard that it actually might be possible with Large v3. I haven't been able to test it yet, but here is how people apparently do it. Set language to Portugese. Turn OFF Translation. Now it apparently should translate any language to Portuguese. Make sure you use Large-v3 or Large-v1. Let me know if it works or if its bogus. Other than that, if a good opensource Translation model gets released, I will add translation to other languages!
Thank you so much for replying! I tested a little bit and that seems to work :D Unfortunately my pc is a bit weak for Large, but I'm happy to know the translation is a thing at least!
I speak English and Swedish and kinda mix the languages when I talk. Can this recoginize when I speak swedish/english and just show everything in english? If it can, is it possible to make it only show caption when it recognize swedish? First one is kinda important, the other is prob just a dream :P
Yes it is very good at combining languages in the way you describe in the first point. The second point is not possible in the current version of the program, sadly.
@patrikzero do you think it will be possible soon to live translate a video (image+audio)? so for example I talk in Italian and I stream with 1 minute delay in english?
Yep, I expect this to come within 1 year (prototypes). I think the biggest bottleneck with this will be Twitch/RU-vid allowing streamers to stream 2 audio tracks for multilingual streams.
Does this have overlay option to print subtitles for yourself on screen? I am thinking of using this to watch movies and auto translate it with the stereo mix mic. Do you know if it would work?
If you use it for movies you could still use OBS, add the VLC (or any other video program) window to OBS, add the overlay as stated in this video. Right click on the scene in OBS and click on Fullscreen Projector. To pass through the Movie audio you can use VoiceMeeter Banana or Virtual Audio cables, but make sure to set it as default communication device.
Hello. We built a broadcast environment on aws cloud server. The cloud server did not recognize the sound driver, so we used a VB-Cable. You made all the necessary settings in the settings file in the OBS settings, PatrikZeros_Subtits_Trial. Now I have to run cmd, but I see an error as shown below. Error occured. 'NoneType' object has no attribute 'close' Press enter to continue... How do I fix this error?
Not by default, one way at the moment is using a Virtual Audio Cable with Voicemeeter Banana. Routing computer audio to the virtual audio cable and using the cable as the default microphone. This has been discussed on my discord so you can find more information there.
I have one questions like If we wearing headphones. Does it show the transcript when I open the youtube or any other kind of platform like the movie on browser or zoom , does it show the transcript ?
Hey ! I bought already but im stuck . I use streamlabs obs but i add to obs but cannot see any subtitles. Can you help me with streamlabs obs ? I see options there
Hi @PatrikZero, I was looking for this for quite some time, thank you for this valuable contribution, I will surely use it soon; It would be very helpful if you share what the minimum specifications are to obtain decent results, both using a single PC and also using two, and also what the minimum specifications would be to obtain optimal performance, it would have been very good if you had included that information directly in the video, it's not hate, what you do is fantastic.
Great idea. However, I use Wavelink. All audio tracks run on a common digitally mixed OBS audio track. So the program would translate everything in the stream and that would be disadvantageous. Is it planned for the future that you can preselect the input source "the mic, or another source" separately in your program settings? Next question, could you imagine offering your program with a fixed price or will it remain with your level 2 monthly support? Best regards
Hey Patrick, would you be able to add the new "Systran/faster-distil-whisper" models to the list of available Whisper models in the next update? It would be awesome
Does it have to be connected to the internet for the 'translator'? (I asked this because I think it is suitable for making videos, apart from live streaming)
Hey, I bought the program but "Settings" one fails to start first with api-ms-win-core-path-l1-1-0.dll then it displays "Error loading Python DLL" then location in Temp followed by "LoadLibrary: PyInstaller: FormatMessageW failed." ...... any chances to make it work? Subtitles program doesn't even open the console properly, just flashes for one frame.
Hello. Thank you so much for such an incredible program! I will definitely use it!! Is there a way to use this in a way that simultaneously shows bilingual subtitles, i.e., where it shows subtitles in the original language while also showing the English-translated subtitles? Or, if it can’t already do this, is there a way to modify the program to where it can do this? Even if it involves sending the audio stream twice to Whisper? Thank you so much!!
Hey! You could in theory just run two instances of this, change the backend and frontend ports in the settings of the second instance and it should work!
@@candlespotlight Did this work? I want to do the same thing. Speak 2 languages, and display captions for the spoken language, and then subtitles for the translated one, and be able to speak either language whenever I want.
Hi. I downloaded the free trail, but the settings in the extracted file is a JSON instead of an application files. and when I run the .exe, it does not work. It does not open the terminal as you have shown in the video tutorial. Any help?
The overlay works for OBS on all platforms, including offline recording! Maybe the OBS Websockets can even send subtitles directly to RU-vid, but I haven't tried that yet, though. The overlay for sure works, so you can have subtitles on any platform!
4:18 According to your warning, your application uses the default windows microphone. If I have voicemetter banana and play with the settings is it possible to translate interviews from other languages in real time while keeping a different microphone line in obs or am I wrong?
If you set up Voicemeeter in a way that will allow you to set its output as Default microphone, then what you are proposing should work 100%. This use case is similar to the use case with Dual PCs, where we transmit audio data to a virtual audio device.
Hello Patrik, I bought the paid version of the tool, and it is an amazing tool! But unfortunately the paid version keeps saying "Thanks for watching" all the time. How can I resolve this?
Hello! I have just now released version 3 of the program. It has a file called "ignoredSentences.txt" In that file you can add sentences like "Thanks for watching" and that will make the program ignore such sentences. This is the only fix possible, as it is a side effect of using a model that is not optimized for real time usage in this way. You can access v3 in the email you received when you purchased the program.
Hey man, this software you made is awesome, I have been looking for something like this for a long time. But I have a question, I play games like dayz where I interact a lot with other players and Im curious if there is a way to make it pickup my voice and game audio(to translate other players) at the same time?
You could use something like VoiceMeeter Banana for this functionality, there you would create a "microphone" which would combine your game audio + your actual microphone. You would set that Virtual Microphone as your main output device in Windows and that should work.
@@PatrikZero no I did not install the cuda driver, I wanted to check how it works with cpu, I guess it wont work without the cuda driver at all? My GPU is a 3070
Hey, so I installed the cuda driver today but it turned out the trial has ended and I cant test it (I have limited data so I had to wait for renewal of my GB) is there a way I can get the trial again or no? If the program works without eating much of my performance I will definetly buy it but I just don't want to spend 10$ if it doesn't if you know what I mean@@PatrikZero
Color palette was replaced by a HEX text input field because the Color palette wasn't working for people in OBS. Also I am sorry but I don't understand the first sentence, could you elaborate about what grey screen you mean?
@@PatrikZero it's grey background full fill the the screen by the OBS SOURCE(not the "background" listed in the settings menu which only around the subtitles)
@@wargamemate Do you have Custom CSS enabled in The Browser Source? It should be something like - body { background-color: rgba(0, 0, 0, 0); margin: 0px auto; overflow: hidden; }
Hi@@PatrikZero , yep I figured that out trying it offline. Fun fact might be that it crashes if you completely disable all NICs. Won't work then. Great app though, it works as supposed to be, besides having the small model just keeping repeating the thank you for watching etc standards.
@@RoyD2 That can be fixed by using V3 and adding all the sentences you dislike to ignoredSentences.txt This phenomenon is caused because we are using models that are not optimized for real time use. When they are reaching the end of a recording, they are biased to think any noise is related to a sentence that would fit an ending of a video. When inputting real-time, we are always close to the end of a "recording".
This is Amazing, how can i have this application take sound from Desktop instead of my mic? e.g If i am listeing to a russian song on youtube, i would like this app to translate that song in english (sub) Is this possible?
Hello! I have just now released version 3 of the program. It has a file called "ignoredSentences.txt" In that file you can add sentences like "Thanks for watching" and that will make the program ignore such sentences. This is the only fix possible, as it is a side effect of using a model that is not optimized for real time usage in this way. You can access v3 in the email you received when you purchased the program.
hi chinese user here. i was wonder why is AI when it recive noise or something it can't recognize. AI always translate to something like "Thanks for watching. see you next time. subscribe and hit like button..." this kind ending sentence. is there any way to stop AI using these sentence? thx
Hello! I have just now released version 3 of the program. It has a file called "ignoredSentences.txt" In that file you can add sentences like "Thanks for watching" and that will make the program ignore such sentences. This is the only fix possible, as it is a side effect of using a model that is not optimized for real time usage in this way. You can access v3 in the email you received when you purchased the program.
I am not familiarized with vMix but if it has a Browser Source it should work fine! Otherwise, you could just open it in a browser window and window capture that.
This is incredible! I noticed there are some translation features in the settings app. Is it possible to speak in English, and for instance print Spanish subtitles by any chance? I can see where I can set the input language, but not an output language.
At the moment only English translation is possible, if a suitable model for translation gets released I'll update it for translation into other languages as well!
@@PatrikZeroThis is a bit unfortunate. The possibility of translating from English to Spanish is what drew me to this. I wouldn't personally call it translation as much as I would real time captioning. If it gains the ability to translate I would love to buy it.
@@divnad You can translate any of the supported languages to English, therefore it is also translation in addition to captioning. If a sufficient open source model gets released for translation to other languages, I will add support.
Hi. I really liked your program and would like to use it on my streamings. But paypal is blocked in Russia, and I can't buy it through ko-fi. Is there any other way to buy it?
You can try it in the Trial, set model to large-v3 and set input language to French and start speaking Russian. Also, make sure "Translate" is off. I can't vouch for the quality, but some people have been reporting that it was decent for some languages.
@@franckywithyou Oh, my bad. You can download a new version of the trial here - drive.google.com/file/d/1DNHcWtHfJi7RU6onQBUGkBkS5FbZLKIF/view?usp=drive_link
@@PatrikZero In the settings i choose large-v3 but under that i just see the option INPUT LANGUAGE but not TARGET LANGUAGE and START SPEAKING as you said :(
@@XMyBrokenPromiseX You need to add "Thanks for watching" to ignoredSentences.txt along with other sentences you'd like to get rid of. In case you have any more questions, feel free to add me on Discord, I'll be able to answer them sooner! My discord is patrikzero
It is definitely possible, I haven't added that feature yet though. If people ask for it in the Feedback Discord I will add it! I could probably save them in an SRT format as well.
Ahoj. Chcem sa len opýtať, lebo mám záujem si tieto titulky urobiť, len ako som čítal, tak to funguje aj na AMD grafické karty. Píšeš na svojom Ko-Fi stránke o tých titulkách, že potrebuješ grafiku GTX 700 a vyššiu, tak či to bude fungovať aj na AMD alebo tam bude treba CUDA jadrá alebo niečo, čo NVIDIA má a AMD nie. Grafickú kartu mám AMD RX 5600 XT. Ďakujem veľmi pekne
Na mojej stránke je, že treba "Nvidia GPU" :/ Žiaľ na AMD to na 99.99% nepôjde cez GPU inferenciu presne kvôli CUDA jadrám, ako píšeš. Cez CPU by ti to samozrejme šlo ale na Slovenčinu by si potreboval aspoň Medium model ktorý by bol na CPU príliš pomalý.
@@fujik2000 If there is a service that provides good free translation for text I will create an interface which will use it for this application, if anyone can recommend a good translator then I'll do it.
@@PatrikZero do you know some payed platform which you would be able to integrate? Honestly, I have currently not so wide overview of platforms which are available to integrate, so just asking.
The trial version is the full version but simply time limited, use the tutorial and it will work as it should. The Speech to Text is easy to access (it is "seeable" in trial) if you follow the tutorial. If there is any error, let me know.
hey, i just got the program but my dualpc setup is a bit strange , i am sending the sound through streaming pc and all together into my goxlr, i would like to traslate not only my microfon but my desktop sound also. i saw that is possible but i dont know how icause i am also using streamlabs . where i can contact you?
You can contact me on discord at PatrikZero With a GoXLR this should be easy, just set your broadcast mix as your default microphone in Windows and launch the program.
