AI agent + Vision = Incredible 

Which vision-enabled agent do you want to see me building? Leave comment and let me know! 🤖

Your content has the right amount of abstraction, making your videos short sweet nd appealing to a wide audience (it's not a course). But at the same time it has the right amount of technical detail for devs and engineers to replicate what you are demonstrating. Thank you for this great content

Hi Jason, yet another great video, I learned a lot from your channel. Thanks for sharing your knowledge.

Hey bro good job on that thumbnail! I didnt even realize it was one of your videos I honestly thought I was clicking on a VICE video lmao,

This is exactly what I needed. Thank you.

I've been needing to see a video like this where someone strings together some ai with code, glad to see. I want to add eye tracking to ar and ai vision. It would be nice to quickly ask questions in the real world. And the automation aspect is very nice for you to share, plus continue to make informative instructional demonstrative amazing videos like this! Thank you!

This channel is real gold! Thank you so much

You're the best. PLEASE POST MORE OFTEN!

I really appreciate you and the videos your creating ai Jason. They are helping me a lot in learning this space

Great content. You have a new sub. Keep up the great work!

Thank you for this clever integration of tools!

Great stuff I was looking for vision autogen tutorials

Thanks for the great content!

Wow this content is great. Subscribed

🎯 Key Takeaways for quick navigation: 00:00 🌐 Introduction to AI Vision Integration - The video begins with an introduction to the integration of AI agents and vision capabilities. - AI agents with vision power can revolutionize various applications, from web design to answering complex questions and enabling general-purpose robots. 02:06 📸 Multimodal Models and Their Potential - Multimodal models can process not only text but also images, audio, and videos, enabling them to understand different types of data and their relationships. - GPT-4 Vision (GPT-4V) can handle various image types, including photographs, text within images, diagrams, tables, and floor plans, unlocking numerous use cases. 04:36 🧠 Understanding GPT-4V's Abilities - GPT-4V demonstrates impressive out-of-the-box performance, such as identifying objects, recognizing people, counting objects, and even understanding perspective. - However, it also has limitations and can make mistakes, particularly in tasks like text extraction and chart interpretation. 06:49 🚀 Promoting GPT-4V's Performance - Different prompting techniques can be used to improve GPT-4V's performance in image-related tasks. - Techniques include providing detailed text instructions, setting performance expectations, using few-shot prompts, and visual referring prompts. 09:19 🌟 Expanding Use Cases with GPT-4V - GPT-4V's ability to understand the relationship between multiple images opens up new possibilities, such as calculating costs from images or determining the sequence of images in a task. - It can also facilitate interactions through visual annotations, allowing users to point or circle objects for AI understanding. 11:44 🤖 Building Autonomous AI Agents - GPT-4V's capabilities make it possible to create autonomous AI agents that can continuously improve image generation and perform tasks like desktop automation. - These agents have potential applications in various industries, from architecture and engineering to customer support and medical diagnosis. Made with HARPA AI

I cannot even begin to imagine the API costs for running things like these on frontier models right now. As impressive as it is, you'll need a real profitable use case if you wanna use it like this.

You are the best!

Notification enabled for that parse automation!

Chatgpt: I can't count apples... Prompt: You are an expert in counting! Chatgpt: Giving the correct answer :) These prompts are just fascinating - and great content as usual!

Always providing incredible content. 👍 👍👍

First! Good tutorial man

Brilliant

Great job, Jason. In the future could you please consider showing how to use these tools without paying for any API keys. For example, using PaLM API or some of the open source models. This is because building projects at scale is cost prohibitive using recursive tools like Autogen.

AI Jason is da man

This is nuts auto gen is a game changer

very good man :))

Thanks mate great stuff would really be interested in agents that can generate video based on prompts even if the agents need to be first trained on sample videos.

how to use a uncensored stable diffusion variant with this. Great vid by the way can't wait for what u do next! Also could it be that the discord invite link is broken? Can't wait to join!

Would that be possible to build a similar agent but improve on illustrated, short stories for kids? That way it would improve both the images as well as the text provided in the stories... BTW, I've been learning so much from you, Jason! Your channel is a gem! As a fellow RU-vidr, you make me feel small...

Really nice video, i for myself would love if the agents could start running stable diffusion on local machine. Any chance you want to create a video about that?

I would love to see one video that summarizes the most popular libraries and api's for llms along with which are the best to work together and which have replaced older ones. Include: AutoGPT, MemGPT, ChromaDB, LangChain, Ollama, Pinecone, etc.

make a video about memgpt

Would be awesome to give the agent the ability to do inpainting!

My Request Please... Can you apply this GPT4V Agent? Simple shed photo and analyze its size, the pitch of the roof, and perhaps how many or how much wood is used to build the simple shed from a photo. Thank you for considering this and testing the multimodal capabilities with this use case.

Hey can u do an agent where it has to do instruction following in a simulator? I think that will be a very practical and interesting application

Hey hi jason it is really good. can you upload browser automation. i am really more interested on it.

can you write agents that operate a headless browser. Within this browser, one window can utilize GPT-4's website features designed for Plus users, while another window can generate images using DALL-E 3. These images can then be uploaded for review in the same headless browser session. Although you'll be limited to 50 prompts every 3 hours, this setup should still be sufficient for most use-cases. Additionally, this approach allows you to conduct user interface analysis or other tasks without incurring API costs.

Now to really test it, see if it can pass the "Are you a robot?" prompts lol.

Create a bot that can read, and visually react to output, so that way it can create a Deep Learning type feedback loop improving upon itself until it no longer can

Ok you know how the game no man's sky you get a analysis visor to scan the environment and save data on plants rocks and animals. Something like that for irl.i would be collecting data all the time

Great video as always. Can you please put your head to the bottom right. It cut off a lot of the content. :)

They need to make it be able to work on less information and make correct deductions that aren't in its training data before trying to make it more generalized. Otherwise it will just compound hyperbolically the amount of information they need to be able to understand as much as a human can.

How about a SuperAgent that can create other agents by its own to perform a complex task?

Can you use this feature with the gpt4 api?

For what i'm trying to do llava isn't yet good enough like GPT-4V. GPT-4V has once again raised the bar for me. And now the waiting begins for an api.

So interesting. Is it possible to get the code to experiment with it?

I think another good video would be comparing these different agent creation frameworks. Feel like I see another one everyday. I specifically would like to hear your opinion on autogen vs superagi

I'm always hitting the token limit after just a few posts of the agent. Is there a way to work around this?

hahaha oh man, you are a legendary.

How do you finetune llava?

I want to be able to use AutoGen or something like that, to set up adversarial agents to use Stable Diffusion for me. So I can ask for an image before I go to bed, and by morning it'll have worked out something.

Our obsolescence as human beings isnt to be feared, but celebrated. Rejoice!

It got the speedometer right - just equated the wrong measurement circle to MPH.

Could you give the source for the file and examples shown in this video ?

Can agents browse and interact with the internet yet?

Full website builder. Full website builder. Full Website Builder

GPT4V will soon be able to take research papers write code, write thesis, get a job, then marry....wait what thats what humans are for....

« 2023 »

there were too many basics in the video, I enjoyed your videos were you provided a more in-depth review like in the Autogen video

I wonder what kind of results you'd get if you were to feed that model some proper English...

`• I believe we are meant to be like Jesus in our hearts and not in our flesh. But be careful of AI, for it is just our flesh and that is it. It knows only things of the flesh (our fleshly desires) and cannot comprehend things of the spirit such as peace of heart (which comes from obeying God's Word). Whereas we are a spirit and we have a soul but live in the body (in the flesh). When you go to bed it is your flesh that sleeps but your spirit never sleeps (otherwise you have died physically) that is why you have dreams. More so, true love that endures and last is a thing of the heart (when I say 'heart', I mean 'spirit'). But fake love, pretentious love, love with expectations, love for classic reasons, love for material reasons and love for selfish reasons that is a thing of our flesh. In the beginning God said let us make man in our own image, according to our likeness. Take note, God is Spirit and God is Love. As Love He is the source of it. We also know that God is Omnipotent, for He creates out of nothing and He has no beginning and has no end. That means, our love is but a shadow of God's Love. True love looks around to see who is in need of your help, your smile, your possessions, your money, your strength, your quality time. Love forgives and forgets. Love wants for others what it wants for itself. Take note, love works in conjunction with other spiritual forces such as faith and patience. We should let the Word of God be the standard of our lives not AI. If not, God will let us face AI on our own and it will cast the truth down to the ground, enslave us and make us worship it. We can only destroy ourselves but with God all things are possible. God knows us better because He is our Creater and He knows our beginning and our end. Our prove text is taken from the book of John 5:31-44, Daniel 7-9, Revelation 13-15, Matthew 24-25 and Luke 21. Let us watch and pray... God bless you as you share this message to others.