AI Leader Reveals The Future of AI AGENTS (LangChain CEO) 

Subscribe to my newsletter for your chance to win a Rabbit R1: gleam.io/qPGLl/newsletter-signup

One of the current holdups with our current lineup of large language models being used as agents is their drive to remain self consistent allows them to hallucinate reasons why they are correct in scenarios where they are incorrect. Having a way to make these models more accurately self critique their own responses will go a long way to improving agentic systems.

Specialized agents seems to me to be a better way to go for now. I would like a security agent who constantly (rt) monitors my state of security both online (personal data, PWs, ransomware, etc.) as well as hardware, and home security. Next a medical/health agent to consult, keep healthcare costs low, schedule appointments, monitor vitals, chart data, etc. A general information/research agent to keep me informed about my interests, hobbies, trips, and creativity and to assist me in these areas. Finally, a financial agent to monitor my investments, complete/submit tax forms, keep taxes low, steer me clear of bad decisions, and to manage my living expenses with recommendations (rt) to lower costs and increase my purchasing power. Perhaps down the road, agents will communicate with each other and eventually combine/overlap until, from my perspective, I see only one agent.

Great video, Matthew. I appreciate your relaxed presentation tone that is also highly illuminating. You are really great to listen to and learn from.

In the very near future everyone's best friend will be AI. In fact you'll have a whole array of 'friends' (aka: 'agents'), that will monitor, remind, plan, suggest, assist and produce just about anything you can imaging. And you'll wonder; - how did I ever get along without it?

These continuous improvements in AI are amazing. Thanks for another great video. Looking forward to seeing the next one!

Make complet tuto about LangGraph and then Langgraph+CrewAI

New ideas are often best encapsulated by giving them names. These names might not be new in that a existing word or phrase might be used to name the new thing but it takes on a new meaning once so named. For example Jail Break or Hallucinations are such new names. So I am adding Rewind to that list in that this is a key concept and the name rewind captures it quite well. So maybe there might soon be a need for an AI dictionary that has all these new names.

This is such a great explanation of the complexity of WHAT an agent actually does. The graphic Harrison uses is so helpful and the deeper dive narration from MIchael makes this a great resource!

I'm new to "flow engineering", and very excited to step up to this level of AI. Lots of value. Thank you.

I believe that at the same way when as humans we reflect on what we are writing down we do go back delete and rewrite, when we have to work on a big task we split it in pieces and we may need to delegate the work so that who is good at a specific task can focus on it. I do not want that embedded in the model because at some stage I will require the model to be Expert in a specific topic and give him access to knowledge that will augment his context and give out the better result possible. I believe that AGentic Workflows will be a thing for a long time.

I'm grateful to you for providing these awesome videos. Your videos are not based on hype. You provide quality, pure-curiosity driven content. This is why this channel is the best. I can follow the journey of AI only by following your videos. This is a privilege. Thank you! Thank you! Thank you!

Awesome video matt. To answer your question, I feel that many of these things are short term hacks. I believe long term the architecture looks much different as we design more modular structures. I believe they will be much more customizable and augmentable.

The problem with building too much agentic capability into the base LLM is that every added feature or component represents a design decision that will ultimately be a compromise between capability, utility and cost. If all of these decisions are made centrally by the developer then the danger is to end up with a bloated and expensive model that fails to meet the exact requirements of any individual use case. I suspect that the future of agents is to develop LLMs with robust but standardized hooks to external agentic components. These components will be available like in an app store. They will be mix and match to meet the needs of any use case.

Langgraph js stategraph simple fast tutorial would be unimaginably valuable IMO.

I would look at them as more of software instances with internal workflows/processes and access to external tools that bring about agentic behavior

I agree with Matthew, we will require a different architecture which will not just depend on LLMs

ultimately we will need to have systems that can do "Active Inference" The Free Energy Principle in Mind, Brain, and Behavior Thomas Parr, Giovanni Pezzulo, and Karl J. Friston

At a basic level, once we can solve NP hard problems (whether through classical computing, or quantum) like TSP, this planning and reasoning will become a lot simpler. There are a thousands of different paths toward a goal, finding the most reliable and efficient will get you a Nobel! Maybe this is what Q* really is! - Until then, we get hacky prompting and 'kinda good' heuristic solutions. Thanks for the video, Matt!

"Sorry, this promotion is not available in your region"

Thanks Matt

I think that idol combo will differ for different types of needs the agent is fulfilling. Sometimes, it remembering too much if my stuff, means I can’t ask it to only take into account my most recent inputs, and therefore it doesn’t give me what I want now, as my own thoughts on the topic have evolved.

Hey Matthew, there are now so many tools mentioned CrewAI, Devika, MemGPT, AutoGen, Langchain etc. pp.. It would be really nice if you could give an overview of how to structure and use which tool in order to build something useful and working. Right now I am quite lost and loosing the curiosity to play around with it. As a full time worker, I don't have much time, but I love to explore this stuff. An consise outlining would really help to know which path to go. Thank you!

agents to me are psudo representation of specific task experts which might embed in a series to steps to ensure they complete there task. This might mean having there responses generated by several different models and then evaluated using different frameworks or other sub-agents. These strategies would then allow us to leverage lots of different specialised models and uncover some of the general shortfalls of using models that are too genrealised. Just like a human, you start wide and narrow in to a problem Doing this iteration processes will enrich the flow and how problems are solved. Basically we need our own mechasims to curate our own infrant AGI models and langchain is the perfect tool to create these abstractions.

Consider the operation of the human brain. It comprises several areas that collaborate smoothly, although this might not be immediately apparent. The Language Center in the brain serves as a rough comparison to present-day LLM artificial intelligence. However, intelligence encompasses far more than this. For instance, there are moments when we can instantly recall answers, such as during a trivia quiz (this kind of thinking is what LLMs are modeling), and other times when we engage in deeper, reflective thinking. We are nearing the development of models that can simulate this deeper level of thought, which will probably benefit from more complex agentic interactions within the models.

I can’t find a link to the tree of thought video in the description?

Nice one Matthew! You are now my inspiration, what a way to self promote 😍

Do you have link to anyone with that released but now removed model?

crewai is pretty rough sledding unless you have an unnatural tolerance for pydantic validation errors and love to read the source code to find out how to actually get anything done

@MatthewBerman: I believe that some model makers will try to integrate Agents into their models but as applications of agents and need for tools grow, Agents will continue to be forefront architecture for designing llm based applications

If the model validates itself we have little or no control over how it does so. By making result validation external developers can use various methods and models to do so

As far as how much human in the loop one might need, this boils down to risk mitigation, and how much of that does one need vs rapid response. As such my former employer had a great definition of risk that was the product of probability of failure (a number between 0 and 1) times consequence of failure, say a number between 0 and 100,000 where 100,000 is very high consequences. So if the prob of failure is high say .2 but the consequence is low, say 5 then the risk would be 1. But if the prob of fail is .001 and the consequ3nc of failure is 50,000 the risk would be 50. Thus the higher the risk the higher the need for a human to check the answer. And the AI could even estimate these as well as employ them.

What I feel is understated is the fact that the more autonomous agents can act, the more they can proactively run in the background and just prompt users when they got something to show, get stuck or need human approval

I think frameworks are going to be needed regardless of LLM development, and human in the loop will be needed for development for the foreseeable future because there are certain things we just understand natively that we don't even think to train AI on.

Is there a link to the langchain video?

Flow engineering is an interesting area of development as I'm reading the book, "Thinking Fast and Slow" by Kahneman, about how our brain has a balancing act of fast and slow processes depending on tasks and goals. I'm afraid if a lot of these processes, like reflection and "think slow," get baked into large LLMs the wrong way, it will hinder flow engineering optimization. "Flow engineers" will be able to optimize the systems they design if they have more specialized LLMs to choose from. So it makes me think there will be markets for more genaneral, do it all LLMs, and a variety of specialized LLMs optimized to do various functions, like parts of the brain or parts of organizational structures, like agents are already doing.

I think LLMs with the ability of thinking slow will not be able to generalize in a way that they have good performance for every use case. I think that algorithms for chaining and evaluating thoughts will be different for different use cases, hence many types of specialized agents will be built.

The way I define AGI, is when the system can self improve. And then we can look at what areas it can self improve. So agents on top of (probably tons of different) LLMs will never go away. But will probably be programmed by AI.

Even if these techniques somehow move over to be more integrated in the models themselves there is still going to be technique that are not, and building systems that apply models is going to continue to be a thing. We need to have a whole new type of models that can implement continuous learning. But there are other benefits from having auxiliary systems, for one telepathic abilities for AI.

Rewind is also very powerfull. Most of the LLMs today stuck at some point. This maybe caused by some misprompting or some wrong word used, and then they stuck. Most of thee time you start again and try to not ask this questions when that happend. Rewind will be powerfull. It is not realy the human way but for a tool very helpful.

I would not know why these techniques would be needed forever. Only if I would assume that LLMs will not get smarter. Besides that the LLMs will get smarter, you can also see big approvements with smaller models, because the training got more efficient. I think there are two processes that advance AI at the same time. One is the LLMs themselfs through more parameters and the second is better training data, better format of it or better quality. What I am looking forward to is when the Model reaches a level and it can actually say that it doesn't know the answer or it has not enough data to answer correctly, instead of just giving a random answer. For me this is one indicator for AGI but I think this could also be done with agents. Basically identifying the quality of an answer or a possible answer. You could than for example give it a book as PDF or other papers, so it's able to provide the right answer.

All of these prompt engineering techniques like CoT, ToT etc are not going to go away, LM’s require next token prediction, so they can only predict the next token after producing a token. however, what we will find is that the outputs from these actions, will be written in the code environment and not presented to the user. The user will see the final output. If it’s not in the code environment, it would be written into some form of “preliminary response” type memory.

Not convinced about human in the loop for long. The missing part is to generate data from this process itself and further train the underlying model. With enough iteration, the human like reasoning or AGI should emerge.

I think LLMS are going to offer different ways of planning and reflection, as an they'll do this naturally but, more interactive say "agile" models are going to emerge. Thus as you say in that regard human in the loop will not go away anytime soon.

No link to the actual talk?

I made a while LangChain program that i offer to local business that want conpany agents. I use the "directory loader" and then use unstructured to load all file types from the directory you chose opon opening. It saves all your history and then re-uploads thats history to the agent database so when you run the app again, it has a memory of your last conversations and "changes" you tell it to make. Ive noticed that it keeps these changes i tell it even from previous days. Not trying to promote but i did make a mediocre video about how to use and install it.

I realized on the UX that its more like figuring out SA (situational awareness) extrapolating to my career in military avionics the faster a pilot can determine an issue/threat was what was needed in is display. I don't here this in the presentation...just that some arranged the best four square that people liked...It would feel better if it was talkes to with SA and the defined A established which may change for each project or agent tool combination

Excellent

hi ...i am a novice researcher ...i want to research in the area of future of agents..can you tell me what are the research article to read to understand this better

These are short term hacks that are useful for a number of use cases when they are reliable and their tasks are bounded set of outcomes. But to make them useful and reliable for complex tasks involving degrees of uncertainty of outcomes, needing strategies to achieve shortest paths to objectives, we’ll need more techniques and technologies.

8:35 short term. I agree with you Matt, I don't think Transformers will be able to do this natively without a serious change in the algo.

So essentially current stages of development are around giving an agent an internal monologue to verify it's own results.

BTW, another key agent might be one that allows one to communicate to the AI not just with words (be they typed or spoken) but graphics and gestures that can be captured on a cell phone camera. And even though I tend to hate having to plug my own videos, I made a video on some ways one could do this and uploaded it to my RU-vid channel which a called YACKS, Your Own Color Keys where one can used colored markers to do this and make ones own color keys to add AI in making sense of a hand drawing, even if it is drawn so as to be rather sloppy and incomplete. And, BTW, I can't remember if I mentioned this in a prior video, so i so my apologies for repeating it here again. But the idea is that human in the loop is important but can be a time limiting factor so the more quickly and easily a human can input info the less of a time impact the human would impart to the process, plus save the human time and effort as well.

My hunch is the network has too resolve the question at hand, so I don't think the model can be designed to have planning built in. But there could be a method to natively feedback without decoding first eg maintain vectors between planning stages. I always thought Open AI did actually always have the rewind step from the beginning. When the first gen of ChatGPT was launched you could see it re-writing the sentence as it optimised the answer. The current version hides this behaviour and no longer streams the output. What I think gPT 3.5 does is back track the layers if it gets a vary low probability for the last token it generated. Then tries a couple times to get I higher probable token.

Flow Engineering is a great term that intuitively makes sense to me.

8:28 pretty certain it's mostly a temporary hack, but some security researchers will still finds some exotic hacks over time.

Recursion/looping plus being fine-tuned to take advantage of that, should be enough IMO. We've already seen what they can do with a little push and some jerryrigging; seems to be in reach to have one that is native to that kind of environment. I mean, of course, not in sealed frozen box, you gotta let them interact with the world and store new information and such. In short, what people that studied AI safety for decades have been warning us not to do... lol, I still remember how common it was for internet randos to just say "keep it locked in a box and we'll be safe"....

I love how he instantly decided to give the rabbit away haha its really useful im sure haha

I think AI Agent can work better with standards such as BPMN 2.0, a business process management standard. Like the famous statisticien W. Edward Deming said, 94% of failure is caused by process, not people. I think Agentic workflow will not be an exception to this rule.

So when is Q* coming out?

These technics are short term hacks I think. Take planning for example. We already know the future llms of openai will be able to do this. As an agent developer you habe to ask yourself if it is worth the time to develop a complex system just to be „steamrolled“ by OpenAI later on. Tools is maybe a little different. But I think it will also be really simple to e.g. connect to apis if the llm gets better and better.

Harrison Chase, CEO of LangChain, highlights the potential of AI agents while acknowledging their current limitations. He emphasizes the importance of going beyond simple prompts and incorporating tools, memory, planning, and actions for agents to be truly effective. He stresses the need for better "flow engineering" to guide agents through complex tasks and suggests that future models might inherently possess planning capabilities. Chase also explores the user experience of agent applications, recognizing the balance between human involvement and automation. He praises the rewind and edit functionality seen in demos like Devon and highlights the significance of both procedural and personalized memory for agents. Overall, the video emphasizes the exciting possibilities of AI agents while acknowledging the ongoing challenges and the need for further development in areas like planning, user experience, and memory management.

@Mathew Berman, why do you need my full DOB to subscribe to your newsletter ? Are you doing segmentation of your subscriber or something?

it's a matter of time before there are enough use cases where there are no longer generalized models, every model is designed for a specific use case, and there is an agent that query routes the initial request to the model that is best suited for the task. Even the query routing agent, will be fine-tuned, to route to the best models and tools for the task I also think hallucinations will be "detectable" and we will likely just see agents asks for more clarity, or something like that. I built a script that does this, it's not an agent really, just sort of appends the clarification to the initial query... but anyway - this is incredibly reliable to design - very simple to build too

When are we going to see an agent that can give a response that includes pictures & diagrams in addition to the written answer? I have uploaded about 100 service manuals which I would like to be able to find information - which often includes pictures and diagrams. Not found anything taht can do that yet.

My dear Matthew, you & your channel are rocking the internet, all the best insh'Allah

These things are short term hacks, but it still play a long term role as competing models provides different value add that call for a integration framework like LangChain to play a unique role.

IMO they prompting is a workaround, the current LLM limititations.

Thank you for speaking naturally in your videos 👍

One of the things humans do when problem solving and planning is to visualize the possibilities. An agent could gain that ability.

after agents comes "neurosymbolic ai". Basically it means adding agents that are not LLMs but are rule based databases, expert systems or symbolic languages. Things like CYC and wolframalpha. Then you end up with an AI that understands physics, engineering and actual reality vs possibility. Agents are going to dominate in 2025 & I believe neurosymbolic Ai is about to blast off dominating in 2026. By then we will be dealing with super intelligence.

hallucination still is a problem today. I prompted in german today 3 top of the line local LLMs with word "Drachenfrucht" meaning Dragonfruit. This is a real fruit an all of the Models made something up with an magic fruit from tales with dragons. Maybe a good question for your tests.

I think that zero shot prompting is the aim of bots, so the prompting technics will not stay forever.

I designed this a year ago these people either need to hire me or catch up. In my Agents code I had the ability to integrate LangChain and did not find it that useful.

Automate the whole process. Have 4 to 6 "master" agents who can spawn numerous sub-agents. None fixed function. Have the master agents decide what function and task they need and assign it to the sub-agents. All powered by AI. Viola - AGI

2:00 that that that

I'm trying to set up my own AI system with Llama 3, Crew AI, and LaVague AI

The next group of people to step onto the AI ​​agent stage will be philosophers

AI Agents: they are a very initial attempt from the open source community to make llm models perform better . For sure, this kind of techniques (what Agents do now) will be embedded into the models with GPT5 or GPT6. I'm sure that the open source community is providing also new ideas to OpenAI and other on how to improve the LLMs (and these ideas will be embedded into the models soon)

JavaTheCup did aomething similaire, I think he usw Albedo as well. He found out, that you can place Geo Constructs before the challange and use the Explosion of auto destruction to destroy some of the targets.

Please keep the Rabbit R1 😅

Its just a matter of time. We only need bigger context windows, cheaper prices, and faster speeds. All of which will inevitably happen from hardware improvements alone. Just gotta let em' cook The tooling is the secretsauce. It'll always be needed. What's better. One gpt-7 AGI, or 10,000 orchestrated in perfect harmony? I'd take the latter any day.

Gpt2 is Gemini 2 🎉

In cryptography, Q* (Q-star) is a security parameter used in cryptographic protocols, particularly in public-compromised. Q* is often used to analyze the security of cryptographic protocols, such as digital signatures, encryption schemes, and key exchange protocols. A higher Q* value indicates a higher level of security, as it means an attacker would need to make a large number of queries to compromise the system. Let me know if you have any further questions!

We have to do them forever and refine them.

There likely is an optimal amount of Agents per system. And the only way to determine this would be trial and error. Let’s say K is optimal number of Agents, then Agents > K wouldn’t produce an increase in “accuracy” (whatever that is for an Agent System).

Yes I think these techniques or short term hacks will be replaced by more intelligent transformers models and that’s when these models will evolve to a whole new level.

But storing "memory" as an external session component isn't really memory is it? It's just regurgitating. Real memory has to live in the context if I understand correctly

I feel like the prompt techniques like chain of thought and such are just current hacks. I think more novel architecture will more have this component built into them from the get go. Somewhere between better algorithms and better data (much higher quality data and much more of that much higher quality data). The system will likely be in essence a mixture of experts like system, but better integrated together into something more resembling a singular consciousness. Now don't get me wrong... these hacks are teaching us things that will get us down this path to better systems. Yet I don't think they are the end solution in and of themselves.

No Tree of Thoughts link

Short term hacks… stick the llm back into your diagram and create an Agentic Plane where your Agents live… (this is kinda like inserting a Management Plane through the ISO Stack, i have a patent for that boondoggle.)

MAMBA architecture is going to unlock all that, and its faster, and uses less compute.

If you think about how human beings would go about solving this problem, what you end up finding is that yes all of these different skills everyone can do but there are people who are highly skilled at doing specific functions. Where I think the future is, is that you will have either a collection of models trained on specific components and you use those models within agent environment, or even a workflow. The other option is you end up with something like mistrial with its eight different models built into one and the ability to call on the different models inside of it bigger model set This is affectively how we’ve solved a number of problems not only as humans where we have teams of people who have specific tasks and find tuned or highly trained and skilled in specific functions But we also applied this to computing. we have multicore CPUs and GPUs but we also have specific hardware that does different functions. The CPU can render imagery but GPU does that far better. Think about the human brain, yes we have one brain but we have different cortexes inside that brain that performed different functions The key reason why I believe we will have a collection of models in workflow as opposed to one single model like our brain is because of the fine tuning required for each area of the task. Take a planning agent for example . If you have a general model for planning, it is not going to be able to have specific area training that enables an effective plan Planning in Sales versus accounting two highly specific areas and domains that have different planning requirements in order to complete a task A general knowledge planning Agent is going to be nowhere as effective as a fine tuned Sales or a fine tune accounting planning agent. Rag doesn’t solve that What rag would do is access the company specific information to assist in planning. What this means is that you may not have large language models for all these different functions. You may have smaller specific models, trained on data to perform that specific task. A planning LM fine tuned For Sales, doesn’t need to know how to write code. So you wouldn’t waste all of the resources to train the foundation model with coding capabilities

wow the rabbit is so bad he's trying to get rid of it already! Come for what they already claimed to have stay for the vapours. I'm hoping they'll have a third website working really slowly by the end of the year. The rate of progress unbelievable

Short term hack but long term will be a different format of auto add in.

try deci coder 1B pls

hmm. Interesting detail: Scientists rumor, that in the human evolution there was a mutation, that caused the lowering of the larynx. This made humans able to communicate far better, which leaded to dramatically improved survival strategies through teams coordination.. Could that today be the same power of Multi-Agent-AI in contrast to zero-shot-prompts?

Sorry, this promotion is not available in your region ???

Is it just me or has anyone else noticed that more ads are appearing in his new videoes

Ai jobloss is the only thing I worry about anymore.