How to build a real-time AI assistant (with voice and vision) 

Sir, your AI voice assistant demo's are one of the most valuable and appreciated youtube videos I have come across. Please keep them coming and also would be great to do a demo with Groq for solving latency issue. You are doing great work man and your students are really appriciating it! Thanks a lot brother!

Incredible video. You're taking your content to the next level. Keep up the good work and thankyou for all you do.

Really cool stuff... I def plan to recreate some of these things along with you when I have a bit more time at my computer. Just a thought, adding screen capture in with this would be pretty cool too to get help with whatever applications you're in... I would imagine you could include both camera and screenshot images in the same context and it should be able to distinguish which you're asking about.. or build a different tool that it can function call for that. Can't wait until we get some slightly more expressive voices as an option like OpenAI teased us with.

You turned your videos to the next level which is pretty amazing. I would like you to ask if is possible, show us how to use open source models in those apps. Thank you in advance.

Great video. Keep these coming.

Amazing content with solid explanation. Thanks very much 👏

very interesting! thank you!

This was a really cool stuff. Thanks for sharing such a quality stuff. I ran it on vscode and it worked. The main problem is the latency. It took like 10s for the conversation. Not sure if it is because of the free account or their websocket API has the issue.

Great video, keep going. Just wonder if you can add barge-in function?

Thank you, It is very interesting.

Fantastic, thank you very much. How would you adapt your code if you have multiple participants?

great video as always. Does this system keep chat history?

obrigado, fantástico o vídeo 👏👏👏👏👏👏👏👏👏

Hi, I am working on creating a closed lan-network, using per to per, and will input a live AI agent, locally stored, getting knowledge from LLM, and wonder if it is possible to have this kind of system then running without using internet?

This is really cool. As someone with python knowledge how would you suggest I get started? Any roadmaps or videos you published?

Great info, thx! Is there a way to use local LLM (like ollama, local AI etc) on this platform instead of openai?

Can be used with google flash? Thanks for super content!❤

I hope we can we use gemini 1.5 pro? I will try to make this changes in old code

from livekit import agents, rtc ImportError: cannot import name 'agents' from 'livekit' (unknown location). I am getting this error.

Hi ..thanks for the nice video.. Can we use local LLM (like ollama) instead of a paid one..?

Hi, this is great. I tried and its working. I tried to introduce my self, then I ask, "Do you know my name ?" the response is "I'm not able to remember past conversations." So, Can we make it remember the conversations, and also "remember" my face ?

Nice men thanks

Hi, great video! How to change the source code that it captures my screen, desktop. Thanks.

sir can you help me why my assistant isnt talking back or nothings happening but its recognising in chat what im saying

This is great!

Got stuck with the API Keys, i'm not sure how to set them

Hola Santiago saludos desde Cuba, acabo de ver en su perfil de Github que es un coterráneo. Su contenido es muy valioso, por aquí un ingeniero de IA tratando de salir adelante en este mundo. Me encantaría poder contactar con usted y ayudarle en algún proyecto.

I wish the code would process the timeline.

so i cannaot code can you make toturial for using ph3 which is free and have vision and also use visper ai to convert text to speech and other free tools so minimizing the cost to completely zero I am a student trying out these stuff and don't wanna pay or don't have money to pay for the API or other things so please make a toturial using all the free and open source tools

Hi , i start test with all your steps and got stuck on the fact that text and audio from the openai api are not transferred to livekit, all requests pass in the terminal , tried many solutions...doesnt work

would love to see how to to implement Deepgram TTS instead of OPenAI !

Santiago, the voices of assistant are only in english? or also in spanish? abrazo crack!

your code generally doesn't run in VScode in windows... some things seem platform dependent unfortunately

now i keep running into troubles when using this code, I would love to be able to discuss this so i can get it fixed i want to implement some features to see if it can work for something else to.

Dangggggggg!!!

could i get this on my smartphone?

Is this functionally any different than your previous video?

Curious about the latency. I noticed that you cut the video after each question (after 19:55), so I am assuming it was a few seconds?

2:38

Has anyone gotten this to work on Windows?

Hey I have a issue with key api livekit its telling me error like its invalid

Yeah, this is cool but not helping. I can open the LiveKit interface but can't find a way to get the agent to connect. API keys all correct

Shame that the hook with videos is start open source and then get people draw into handling supporting functions via a commercial platform (for $$$)