In-depth rational breakdowns are missing in this space… Dan definitely is one of the best AI channels on RU-vid.. worthy sub too because I wanna see this channel grow
It would be interesting to explore the evaluator function in depth. Like it’s one thing if you have a hard defined set of criteria for what a good end respons is. It’s another thing when the definition of a good response can be more less easy to define. Like the advice of an AI therapist. If you have three chains you’re getting three slightly different answers and there in general very similar but in that case how do you determine the best answer? My thinking is that in that case it’s really important to understand the end user, in order to be able to accurately choose the best response to deliver them. So it’s the idea that the current mental and emotional state of the end user (patient) and their goals help make the decision of what response that evaluator will choose. It’s the idea that one of those responses will get that patient closer to their goals factoring in their current emotional and mental state. So it would definitely need to probably send the responses from the different chains and the patient data to another set of chains which attempt to simulate the future or predict make predictions.
Yeah I think the 'evaluator' is going to be a topic we touch on again for reasons you mention. "My thinking is that in that case it’s really important to understand the end user..." This is the key of a great prompt chain, evaluator, and really a great product. Determining the best answer (prompt result) comes down to knowing your user and knowing the outcome you're looking for.
Thanks for creating. Helps out a lot ! Just one small question to put an audio low-cut around 100hz because the typing sounds rumbles up the low-end. Keep grinding, good work.
8:10 U could develop system of having the best prompts chosen by an initiator agent and an evaluator agent. Have the initiator agent built into a reinforcement algorithm that's been trained or tuned with a related dataset. Then have both agents run in a feedback loop until the optimal prompt flows is chosen. Have the evaluator use some kind of metrics or benchmarks to rate the output, until the most optimized output is chosen. (This also might put prompt engineers, out of work if it ever became a thing)
I think your fusion is missing a key step in between where each LLM gets a revision step and access to all answers to revise their previous answer. Add that in before the final resolver and you have gold. I can link a demo.
interesting, would love to see this added as an option to Aider. Not sure how one can say this would outperform gpt-5 without trying it, but using gpt-5 in a fusion chain makes sense. Of course though, there's the cost... once the end result is funnelled back to the start does the whole chain run again?
It would be interesting but I also love how lightweight Aider is. It makes few assumptions and relies on the model to do the hard work. It's almost guaranteed though that a prompt chain and fusion chain would increase results coming out of Aider and really any LLM/Prompt reliant tool. In a self improving chain/workflow thats the idea - some feedback from the end result gets funneled back into the start of the chain.
How might we fold DSPY into this concept. Stanford storm employs DSPY. Storm takes prompt chains, so DSPY takes prompt chains. Is there a better workflow?
Vst am int -- Oriental where, he used, policy Dutch Waffen! V=verified -- hence; I served - with them! Hence; I -- had - that int! Fused index! You have, no f(x) - and call it, any int!