Wanted to get to this for quite a while but it was never a priority. Love that I am now able to simply follow your advice instead of going through this all on my own! Thanks for the great videos, they are an incredible resource!
Over the weekend I was working on this exact problem. I couldn’t say which part I was missing, probably different parts at different times. What I can say is how epic it was to start the day with this video, thank you 🙏 legend! … I don’t suppose you feel like taking a crack at autogen next 😅 In any case thanks again!
Great video! Do you know why this doesn't work with a GET request? If I send a GET request instead of a Post (of course I adapted the API) it loads the whole message first and then sends it to the client instead of doing it async.
Thanks for the awesome walk through! Have you had a chance to test this flow with OpenAI Function Calls? I have a similar implementation for streaming, but once the LLM triggers an OpenAI Function Call, it fails to stream - It seems like the iterator just doesn't return any tokens.
Why is it you can create a fairly simple generator to the frontend using the LLM, but when you add an agent, it becomes so needlessly cumbersome? Anyway, amazing job with the tutorial!!! You're a Godsend.
This is amazing video where we can get lot of information . Can you make a video on how we connect streaming with llmchain and memory and show the streaming in webpage or html or streamlit?
hey, when I use tool, the streaming stops. When no tool is used, streaming occurs as expected but when a tool is called, the final response after taking observation into consideration doesnt comes in aiter() function
Great content. It works with Agent, in my case I need to use AgentExecutor instead of Agent, agent_executor = AgentExecutor(agent=agent_chain, memory=memory, verbose=True, tools=tool_list,return_intermediate_steps=False) Look like AgentExecutor is not streaming with LCEL. Any ideas?
I have the same issue. I have a few tools, and an Agent executor which handles all the calls. I am using a SQL database tool, and even though the tool itself is giving me streamed results, the final answer is not streamed. Any idea how to tackle this?
Looks like since the last langchain update all the current calls are deprecated. Do you have any details on how to update the code to work with all the changes?
great video! but whenever i try to run uvicorn main:app --reload in the terminal, i get this error: TypeError: multiple bases have instance lay-out conflict
It seems there's some kind of issue when trying to perform the get_stream("Hi there") the second time. The first time i receive correctly: { "action": "Final Answer", "action_input": "Hello! How can I assist you today?" } the second time i just receive: Hello! How can I assist you today? and that generates an exception
Good example but the result still very summarised specially when using search engine like serper, so the question here how can control size of response text, because there are same cases need response contain more of details.
Thank you very much James for this tutorial! The FastAPI template works perpectly without adding tools. When I add LLMMathChain tool to the agent, the application starts normally but it just stucks at the step "Entering new LLMMathChain chain..." I have only access to AzureOpenAI. I wonder if it is an AzureOpenAI problem or it is a general problem. Have you tried to add tools for the Agent in the FastAPI code and did it work? I also tested with Zero-shot agent, the result is the same. > Entering new AgentExecutor chain... ```json { "action": "Calculator", "action_input": "50 * 48" } ``` > Entering new LLMMathChain chain... Many thanks in advance!
No it’s just a case of modifying the async iteration logic to handle how you should deal with tools (if-else statements to handle when to stream and when not to) - there may be some other LLM math specific behaviour that I haven’t come across though as I have not used that specific tool with this
@@jamesbriggs Ya, most tools should have an _arun method that works with the acall agent function. if it cant run async, you'll have to modify _arun to just return self._run, and it should work. This means you won't be able to stream the result of that tool right away, but you can stream the resulting response from the AI
So I got a question, do we need to specify an input and output parser using and agent for the APIs I'm trying to wire one up to Django rest api and get constant errors saying LLM output can't parse but yet works perfect in Colan? plus is there a difference. betwee other agents and using initialize_agent its hard to get a clear answer answer on the data structures and usage of agent methods in langchain or just a something that is clear and not all over the the place info overload with their docs? thanks, Awesome content by the way. Do you have any other videos clarifying agents?
If not using streaming you can use the standard output parser classes in LangChain, with streaming you should try and parse live, which is harder but typically doable unless the model really messes up I have a YT course on LangChain, chapters 7-9 focus on agents :) ru-vid.com/group/PLIUOU7oqGTLieV9uTIFMm6_4PXg-hlN6F
Question - I followed along and ensured the code matches, however I am unable to emulate this streaming behavior when calling the "get_stream" function. It seems to still wait for the chain to complete gathering the text and then printing it all at once. Any pointers on what might have gone wrong?
I pulled up the sample to check and ran into the issue too. In my working code, the equivalent to the demo `AsyncCallbackHandler`, I’m overriding `__init__` and calling `super().__init__()`. Changing that got it working for me. I bet if you’d used the built-in handler and switched to custom without full shutdown, you wouldn’t catch the issue.
Hmm thanks for the idea@@zacboyles1396 , however it still doesn't seem to work for me - I did confirm websockets and the uvicorn extras as also installed. The standard handler does seem to work better (although not perfect as seen in the video)
I am using flask and html. I add this callback and get the streaming response in terminal but not frontend. I also use websocket, SSE client but not succeed.
I have a question -> What encoding should I use to encode Polish characters, because line.decode("utf-8") returns errors for Polsih letters. I tried other encodings but its, not working. Any ideas @jamesbriggs ?
