Google Releases AI AGENT BUILDER! 🤖 Worth The Wait? 

@19:25 you set your Tool Type to OpenAPI, and the endpoints in the instructions will work. I imagine it's 1 function per tool for modularity. It's more or less exactly the same setup as the Custom GPTs configuration.

Hey Matt. When you select "function", you're responsible for writing the code and driving the conversation via API calls. The flow goes a little like this. You submit a message and you check the status and it tells you you need to run the function with these inputs. YOU then submit the output back to vertex via the API, where it uses the function output to generate the next message. You can implement the function wherever you're DRIVING the convo. Makes sense? This is a very similar flow to the assistants API which I have a video about on my channel if you're interested

You’re so right Matt on that agents in the Mercedes. There’s this weird movement of bells and whistles, and what we can do rather than true functionality that is efficient. Especially when it’s not level five self driving, we still have to drive and less distraction equals more attention to the road and thus is safer driver.

@26:10 they're using the equivalent of Agent (preview) app in the Agent Builder service which as they word it "can answer questions from data, connect with business systems through tools"

Matt on it! Thanks for going over this. Nobody else is really going into it like you.!👍👏

I've been eagerly awaiting a local AI solution to sort through our family's extensive collection of decades-old photos. This AI should be capable of identifying and eliminating duplicate or blurry images, learning to recognize faces, and categorizing photos by individual. Moreover, it should have the ability to estimate the age range of each person depicted. Ideally, this solution would allow for easy exportation of organized data so that Stable Diffusion models could learn from it. Additionally, integrating facial recognition capabilities into home security cameras would streamline access control, eliminating the need for physical keys.

Some of Matt’s videos were a bit lightweight in the past, but he really nailed this one. 👍 good analysis, brought in external knowledge and not just AI fanboy content. Nice job. Will be subscribing to this channel now!

thanks for the review . I was short on time today , perfect timing

Agent > Example > create example The example is a dummy dialogue. You need to actually mockup a weather convo. Each prompt/return is an ‘action’ in the example convo. There are other actions, like tool. Tool is where you put the code

I'll be ready for Google to put that kind of tech to use on youtube. No more trap videos where the title, description, and thumbnail don't match the content. RU-vid could auto-down-vote videos that don't match their title, description, and thumbnail. I can imagine that Google will be the king of multi-modality due to RU-vid.

Considering Google ought to be a solid year or so behind OpenAI, despite their best efforts, what kind of monster is OpenAI planning to release that beats all this?

Hey Matt, with all of these models and apps along with their updates, it would be nice to see a real time list of your favorites. For example you said Gemma wasn't to impressive but it may be now with its new update. Maybe like a place where you can rank the best models or frameworks and update the source as things change. For example crew Ai, autogen, Gemma,, Devin, Devin alternatives, and all the different llms. It be nice to see an up to date list of your rankings in each category. Or does something like this already exist?

3:00 that’s great , I hope the model works well though

#Using ai to select ai, #Using ai to debug code and create insertion points (~cross cutting)....it could then analyze and respond by altering the code to fit conditions (~FPGLA). I'm working on knitting Neo4J and ai-models for an interactive teal time graph dashwindow to use user priveldged based access. It seems that ai could refine alternative models based upon a user's project and use rag to learn preferences...and present them ranked with pros and cons of each.

10:05 GM is already there. Every modern GM vehicle starting from the Cadillac Lyric, 2024 Acadia & Traverse, 2025 Enclave, all have the Google agent built in. As the agent improves, the software will be updated over the air. GM is even phasing out Apple CarPlay and Android Auto, because the Google voice agent will fulfill all the requirements, including infotainment and HVAC control, vehicle information query, mapping, POI search and navigation, traffic updates, etc. Eventually all GM models will get the Google interface.

i heard the keynote before, and I really appreciate your insights and commentary to make it more "real." There is often a strong corporate biase when we hear these kinds of releases. Great video very informative. Love your stuff

Nice video Matt, your huge efforts are highly appreciated. Thanks for sharing!

It's like Autogen with frosting on top. I can dig it saving me a lot of design setup time for agent clouds.

Thanks for sticking through it - I dropped out on the live original presentation when I saw all the customer agent stuff vs the more interesting However, you pulled out a few more things that I missed

Great announcement, thanks for sharing! I've been working with several of these offerings over the past few months, developing some great use cases. I was thinking of perhaps showcasing some of them on my channel soon.

Nice video! Will you cover Mixtral 8*22b soon? That would be great to see. The 8*7b was already great.

Still watching but as always love the content and the way you deliver it, so glad I stuck around as you are my go to to keep up with whats going on

Thank you. Your commentary adds real value to this video. I'll definitely watch more.

Thank you Matthew for all of your amazing videos

Love the analysis and call-outs.

the Mercedes missed opportunity is probably an insurance issue. They'd need to long term test of it's safe because they don't want to get sued if someone expects the ai to read their mind and then goes road rage and blames it on them

Gold video ❤ Thank you so much 🎉🎉

Ok, so I'm not a computer programmer, but I do have something of an interest in A.I for many reasons. One of them being that as an artist, it interests me in what they can do. The other day, I was watching a guy creating agents and without any programming needed. He simply typed what the task of the agent was, and it did the rest. He even connected 4 of them together to different tasks but then combining the outcome to achieve his goal. The way I understand them to work, although I could be wrong, is that you give each a task to perform, and that is what it does. Then, we can share and combine information between them if needed and also its possible to have another one keeping an eye so to speak on each agent and test them should you wish. If I'm correct in my understanding then it's clear by the Google video that they are using them in such a limited way because I can see so many interesting and new uses for them, even with things that have never been done before. Of course, I'm sure eventually you'll be able to do all this just by speaking to it rather than typing. Typing is so outdated. An interesting video, and I look forward to seeing what gets created using agents in the future.

Thanks Matt for the review, as always you have perfect timing and bring a common sense perspective.

I would think the context window is super cool for training robots, cause you could have a video feed going to the llm, or even just training it on tasks, like show it a video of what you want being done and the llm can translate that into language.

Great video. I appreciate what you do.

The tool definition is to avoid running arbitrary code on their server. If every tool *must* be implemented as an HTTP API then they can handle them all in exactly the same way and handle things like timeouts and errors in a standard manner.

Hey Matt. Have you seen platforms like Kisai that do a lot of the heavy lifting with getting from design to production code? Lots of Gen work in there that makes this seem a bit rudimentary.

Would love for you to do an in depth video in Gemini 1.5 pro 🙏

Seems fine, but I'm not switching from OAI just yet. I've got too much built with them to walk away.

DF and DFCX are not previous, they are still current and are now more mature. Vertex agent builder is different. When you need a mix of determinism (ala decision tree), entity extraction and fallback LLm, dialog flow it is. Plus it supports call campaigns, testing, agent assist and all of those things as one click features.

It seems like the Google's assistant functions may be similar to OpenAI's assistant functions. Basically it allows you to observe the response you get from the assistant and call a function in your own code base. No code in the assistant itself for the functions. I haven't used them, but I'd love to see a solid use case for implementing them.

Nice and honest review

Thanks to our sponsor - every news release

Love the analysis

10:24 Note on Mercedes: Every new car has „Hey Mercedes“ - an assistant helping you out basically with everything. And it works for me.

Thanks!

@Matthew; 33:40, you sure as hell will when your shit don't run after the change(s)! cheers brother!

Very interesting analysis

Matt, you mentioned better use case for the agent approach. It would have been awesome if you would have comment after the keynote on each like you did on Mercedes-Benz.

Hey Matt, What's the ticker thing on your bookshelf?

I think you're spot on about Google "playing it safe." They've got this super-global brand; as soon as they make an agent that says or does something questionable, it's huge news. Microsoft figured out how to solve this problem decades ago: let small companies do innovative things, then buy the successful ones.

Tried to setup crewai today and was not able to handle my use case of sending image to gpt vision and get output from it. Any suggestions?

They just don't got that yet. In this case, you are way ahead of them. Keep on 😊👍

The thing that seems to confuse people is actually a very good engineering principle for decoupling systems so that they are not tightly coupled with each other. The language models are only responsible for constructing the input parameters for YOUR code, you take those input parameters and pass them into your code and submit the result from YOUR code (running anywhere) to the vertex AI. You are not bound to upload your code to the vertex AI, it can be running on you local computer or anywhere.

@11:30 - I believe this is intelligent of Google to enable others to first launch their own AI Agents. I'm sure they have many of their own ideas for future capabilities in the making. Incredible tech. Thanks for the share.

I want something that doesn't involve having to use a custom IDE or decrypt a starship console. Is this better than crewAI?

#Google/Agent builder It seems very sales oriented....I believe we all need personal agents to cope with other agents and the information, and data flows.

LMAO the new Mercedes looking like a sentra

The only thing I took from this is that they have optimized their chat sales bots to allow even small businesses to train them without tons of data etc. pretty small innovation, but they need this information pushed out there hard to allow companies to know the service exists as easy as it is.

Agents that can do knowledge work is the real juicy technology.

You are awesome, sir!

Now, i have everything, can someone just give me a good startup idea as well? 😂😂

Agree on voice for car

I think one of the reasons we don't see strong agent use is due to limitations from inherent statelessness of inference. Current ai uses the chat history to 'seem' continuous, but do not retain unsaid thoughts/intent/goals. This causes issues in Long Term Planning, having to rely on low information density hacks like reason aloud your actions, goals, actions; but if any text is missed in the Chat History then the next inference may not be able to infer that missing context. Those internal goals, intent, reasons, thoughts do not survive inference to inform the next inference if not explicitly said aloud and current tech has no way of getting that dense representations across inference -- once the inference ends it's practically a new ai given the chat history up to that point (no previous inference's 'thoughts'); hence the discontinuity causes issues. possible solutions: ~memory-informed inferences (as a vehicle to get dense context across to next inference) ~state based system like a giant Mamba/Jamba perhaps

In my opinion, it is the improvement of search engines to become faster and using the web scrap .

I believe Mercedes didn't mention the most desired use case because the in-car assistant would need to function offline. Google's Agent technology has not addressed this scenario, even though their open LLM gemma 2B has the capability to operate on a car's infotainment system.

If agent can isolate background noise, and enhance voice quality in interns of tone and sound within a norm that would be wonderful, because lots of support staffs are working from home and diverse background.

You were too impatient with the red-top lady use case 😀. It seemed nice

So true Mathew. Refactoring the message around shopping assistants as new A.I. is a sad day for the org that saved search from a flood of spam. How the mighty can fall.

33:40 one helluva stretch of the imagination!

Google is building back their "moat". Very interesting to see the giants battle it out with AI. Hoping to catch some of their scraps!

Regarding your questions- On the tools and API - it works like GPT plugins - you provide your backend api spec using API, and the agent will trigger that endpoint if it wants to. It’s on you to implement the backend logic, and the Ai will just call it.

Doesn't Zapier do the email automation though ?

Cool!

Totally agree

It's enterprise ready if not futuristic. The compliance controls, frozen model in tenant, seems huge when I am freelancing for big corps.

Advanced enough to be omnipresent in the marketplace but not too advanced to make your company obsolete. This is priority number one for companies

The AI madness era🤔

When I hear agents and AI together, I can't help but think of agents from The Matrix, lol. Thanks for the great video and summary to clear that up mate.

Nice video. Same conclusion: Employee Agents seems to be just the Google version of MS Copilot. Are we all missing something or what?

You mentioned Dialog Flow, is it worth taking the time to learn CCAI CX or will AI Agent builders and Vertex supercede take the place? I want to do conversational AI and meet my customers where they're at. Dialog Flow looked a good option, is that still true today?

You have “Hey, Mercedes” - it needs a lot of improvement…but they are working on it!!

I thought about what you said, that Google is playing it safe, but I think what it wants is to bring AI to everyone, and that is not easy at the infrastructure level, I think we have to do that step by step, so as not to falling into the attempt, and thinking like a common user, an AI assistant for me is the best option, which is what we already do with chatgpt, I use it for everything, even cooking, but I pay for it directly

Verbiage and jargon. I thought that the various AI products would separate the task and send it off to specialist agents to work on, An example. Task: "Create a business plan for me to take possession of my very own 200km long O'Neil cylinder at L4." Or Task: "Destroy Einstein's Bent Nothingness conjecture an use the newly discovered physics to design and build an anti gravity craft,"

How do I access this? I've been searching everywhere but I literally cannot find even 1 of these features. Is it not available in Europe?

30:00 pretty sure they can already create entire codebases based on a figma design. (Even create figma designs based on a design system -- e.g. Material UI) Right now they're telling us it's "helping us" but it won't be long before the entire process is fully automated. Brb, learning how to farm.

Wow, google managed to make actual humans sound more like an AI now than actual AIs.

I'd have to agree AND disagree with your take @10:50. True, most of these chat functions are basically interactive FAQs vs a "virtual' CS rep that can look up your account profile. But for the latter, I'd say maturing that level of capability would definitely be cutting edge -- this is a very hard task to execute for a large organization. The range of customer's issues can be significantly broad, and training a model to provide dynamic and accurate responses against a proprietary knowledge base has proved difficult. So much money would be saved to free up resources for other efforts. Then lets also think, imagine the day AI CS reps reaches maturity. Then imagine, we all get our own personal CS rep (looking at you Siri, Alexa, Google). What is the reason to call a company anymore? What is even the reason to visit their website anymore? Or download an app? Every single interaction could be done through our agents interacting with each other. A whole new world.

31:00 this is their new IDE in the browser.

Can someone help me figure out where the heck this "Agent Console" is? I am having the hardest time finding this. I've found the "Agent Builder" but this does not appear to be the same thing.

Can you compare gemini code assistant vs copilot? Some deep testing because I didn't find it that good.

CrewAI still the right way

Kinematics for sports agents?

Please do a video that shows how to access those features within Google. It looks to me like vapourware when I have a look at what we have available.

I hope it won't get discontinued. Like most of the google products

I like the Mercedes @digital experience” when it really is data gathering, and digital surveillance😂 Functionality of a device always has to take precedent. A car is . Point A to point B

Probably offers a little improvement in the performance but I do not see how it is any different from passing an intial prompt and then starting the session

Hey Mathew, did you make any video on a web app that prompts gemini vision pro with image and some fixed text prompts and get the output displayed on the UI. if you did , please share the link and if not please make one.

When it comes to a personal AI agent, I would NEVER allow any government or corporation have tabs on it for what should be obvious reason

I tried geminis the pro version and it doesn't work like chatgpt or mistral and the others.

Matt I completely agree, this whole showing was very unimpressive

I hope it is ok for me to post a question. Do I need to pay in order to use Google Assistant to review my email like the benefits example in the video? Do I need Google Workspace to do that? I think that is a paid version. What about Google vertex? I hope I am spelling it correctly.

It’s not that Google is playing safe, but their business is the data. So they’re moving fast to work with the highest spending advertisers because they’ll have all customer spending data routed thru their ai servers.

Great content and thanks for the hard work. This may be out of your knowledge but can you please do a video on why these companies are offering so many great features for little or nothing. We have seen that there is always a quotation marks "you are the product" type of language interceptions of these features for example in the Mercedes Benz hey Google collaborations. Many have complained that the OnStar incredible features turn out to be that they were selling data to insurance companies leading to cancellation are much higher insurance rates in the future, as they were basically spying on the drivers of their cars. The EU has incredible protection policies that really need to be implemented into all AI products