Thanks so much for this video... Objectively, it's surprising to me that the one thing LLMs seem to struggle with is something seemingly so basic as "memory." I know there is logic behind it, of course, but it's just the last thing I would have guessed would be a common challenge to overcome. These 7 to 70 BILLION parameter models struggle to find a way to remember a single Gig of text, lol. I'm really hoping they make advances in this area soon because, like you, the long-term memory aspect of an AI character is one of the most important aspects to me. Anyway, thanks for making this video... I subscribed and wanted to comment to support your channel. I'll keep an eye on your content in hopes you have a new video about this topic as things progress. For now, it sounds like the Smart Context feature is not working in ST, so I'll plan to let things evolve until long-term memory becomes more accessible and hopefully a more standard feature. Cheers!
have you considered AnythingLLM for your context and memory management? it would also give your AI RAG, long term memory, and agency to function call, search the internet and all the other things.. i was considering it for my build since it also hosts your model at the same time. not sure how integrated it is but i feel you can probably just plug and play it.. might be missing something about the operation of smart context but this reaally changes things if we can rig this together
Hi there @emandc1 After hearing about your suggestions, I went about to read up and tried myself AnythingLLM. From my trial, definatly the user friendliness is there, alot of features are simplified for usage, which is great. Yet, i was not able to get the vector database (default LanceDB) to work sadly. Tried to follow the few video tutorial on youtube but didn't succeeded. Note that adding a file to vectorize and query works great. But i wanted to make it "remember" my facts / conversation, which didn't work for me for vectorization, it seems to only work for that chat history (at max 45 chat logs). As I continued my tests, i notice a few things that was a showstopper for me. Note that i do find this tool to be great, yet, since its still a very new tool (version 1.5 from this writing), it's missing some features that are critical to me. Here are a few: 1. Limit of history context: it only allows for 45 chat history context. It won't let me go above that. Since my chat can be very long, I find this to be problematic. Maybe future version will remove this limit. Truthfully most daily chats i do with my AI does not go above 45 chat logs, so not the biggest issue here. 2. Memory chunks: limited to 1000 characters maximum, and limited to 12 chunks the maximum allowed retrieval from memory. My default is much bigger than that in my workflow (~2000 characters chunks, and 15-20 chunks). I have a GPU that allows for such bigger context, hence, not having that flexibility is a big issue for me. Size and amount of memory retrieval is what allows the long term memories and to allow the AI to "know" about information in the past. Limiting this basically limits how much the AI knows about you. 3. Another deal breaker for me is the inspection of the prompt and what the context is for each reply. It's ok that it does not show it by default for general users and ease of use, but when I want to debug if the "memories" were correctly used, or of the context is good or not etc, then i want to see what was the full prompt + context that was used. In this case, i can't see it, which is bad for me. You can see what I mean by reply instrospect at 8:00 in my video. Again, all this is based on my personal use case only, other use cases might be great for AnythingLLM. I will want to reevaluate AnythingLLM later maybe at version 2.0 or something like that (or if someone knows how to get this to work properly with long term memories similar to what I'm looking here, please share as i might have missed something). TLDR: Review of AnythingLLM for long term memory Local LLM: Yes Very long term memory: No Evolve with time: No Will try again when AnythingLLM 2.0+ arrives (or someone show me how to get this to work for very long term memory)
Hi there. Thanks for your inquiry. After some thoughts about how to support this in the future knowing that the ST Extra is deprecated, from all the tests I've done, we can use the Data Bank and Vector Storage (integrated in ST by default) to do the same thing. The only feature missing would be to allow it to cut at a newline, instead of cutting randomly a given text at a char count, which is what the data bank is supporting right now and the only way to control the cutoff point. I've just made a feature request to SillyTavern devs. If the feature gets implemented, I can share a new video of how to do it in the newer way. github.com/SillyTavern/SillyTavern/issues/2625 Thanks!
I think this is a brilliant way to maintain context and memory, with the summaries. I do have a question, however. Won't your prompt fill up all available tokens eventually, with the summaries added in? Even with a context of 32k, I imagine you'd fill it up after a few weeks of daily use.
Hi there. Indeed, the summary is part of the solution to long term memory: to get data that is not "taken out of context" so that it is still relatable much later on. But they real key to long term memory (so that, like you mention, you don't end up filling your 32k context) is the VectorDB / Smart Context feature. If you look at 12 minute into the video, you will get more info about how this works. But just very simply, all your data summarized are in a big big text file (think 1000k +), but when you write something in the prompt, the Smart Context feature will go "check" for any summaries that is close to what you are writing in the prompt. Note that we're not talking about trigger words here like Lorebooks, but rather a great way to associate words compares using vectors. Hence, once the matches are found (which you can configure how many match you want it to find), then it will only take those matches and add it you your main context (of 32k). In the video, it showed that it took both "summaries" that i created, but in my current data file, i have more than 60 of those and increasing. Each prompt will only go get about 20 out of those. What this means, is that you could have "infinite" long term memory here, since it does not matter how big is your main data file, in the end, the Smart Context will only grab what is matches up to your defined limit. Then again, you can see here that not enough matches will then hinder on the response quality, same for the summaries that we do, if it's too summarized and missing too much details, then what it remembers is also as limited. On another way to see this, even if you only had 4k or 8k context, you could still get infinite long term memory... its just upon each prompt... it will only remember 1-2 things from the past since that will be the maximum of context space you can afford....where if you had 32k context space, it would remember 20-40 things from the past. I hope this helped understand better this long term memory strategy.
Batch Automation File code (just copy this content into a text file and rename it to replace.bat): Batch script that takes the content from tempMem.txt file (make sure there is one), removes all newlines, add today's date, then add the content to the file chatMemAll.txt @echo off setlocal EnableDelayedExpansion rem Initialize the output line set "line=" rem Catenate all file lines in the same variable separated by "|" for /F "delims=" %%a in (tempMem.txt) do set "line=!line!|%%a" rem write out the date set day=%date:~7,2% set month=%date:~4,2% set year=%date:~10,4% rem Show final line, removing the leading "|" echo Memory date: Year=%year% Month=%month% Day=%day%. !line:~1!>>chatMem%year%%month%%day%.txt echo Memory date: Year=%year% Month=%month% Day=%day%. !line:~1!>>chatMemAll.txt rem Keeping the daily entry in raw format echo Memory date: Year=%year% Month=%month% Day=%day%.>>chatraw.txt type tempMem.txt >> chatraw.txt
Can you make a tutorial how to install it? I tried but it says I have to install the chromaDB module... I cant get it running.... thanks for the video!
Hi Stephan :) Happy to hear you want to try it out. Installing these things usually ends up with some trial and errors, since not eveyrone has the same setup, same environemnt etc which ends up with different kind of issues that I don't even dare to try to cover. Best guidance I can give you is to check the smart context instructions here : docs.sillytavern.app/extensions/smart-context/, make sure you don't miss any steps in that installation instructions. If some steps has error / issues, then do a search for that specific error or issue, then hopefully you will find some solutions.
@@AIArtistryAtelierThank you! I think I installed the chromaDB in the wrong path. So all these things need to run locally? I am using runpod for hostig the ooba, to get more fluent answers. BTW the llm you showed is really good. It worked really good with my characters. Thank you!
Quick replies Sumtrait: the quick reply that summarizes the whole chat Pause your roleplay. List of characteristics: Generosity, Humor, Curiosity, Adaptability, Authenticity, Positivity, Creativity, Compassion, Integrity, Respect, Responsibility, Justice, Courage, Empathy {{user}} is interested in developing a long-term memory system that will help him keep track of {{char}}'s core personality traits and {{char}}'s core values. Based on current chat history of events and discussions, update {{char}}'s personality traits score on a scale of 0 to 10 from that list of 14 characteristics that define her core personality traits. A score of 0 is the minimum, a score of 10 is the maximum. Keep it as short as possible without any extra explanations. Use the following as an example of the formatting: Generosity 10, Humor 10, Curiosity 10…
Quick replies Summe: the quick reply that summarizes the whole chat Pause your roleplay. {{user}} is interested in developing a long-term memory system that will help him keep track of events and states. The goal is to write a detailed report of about three hundred words of the events that have happened today so far in the chat history, focusing on major events and each character's associated emotions. The report should be concise and factual, relying solely on information from the chat history and avoiding any speculation or guesswork. Do not use words that is not present in the chat history. The report, restricted only from chat information, should include a list of main characters, events and each character's associated emotions, all of which must be directly related to {{user}} or have potential for development in further stories. Do not give extra details or description if not specified in the chat. Describe seperately for each character, how they are feeling in great details during event list descriptions. Unique words in this chat must be used throughout the whole summary and not just a list. Do no add events or objects in the summary if not infered from the chat history. Limit the events and each character's associated emotions list to ten only. At the end, summarize in details with at least one hundred words what was the most meaningful topic that was discussed today.
Hey i just found this amazing video but i had some problems getting smart context in the first place. I read a couple of the comments but to no avail I've tried a couple things but i just cant get it working, the closest i got was the option to turn smart context on being grayed out I'm at a lost. i noticed you mentioned below that vector storage is the same thing will there be a tutorial using that instead in the future?
Yeah, I think, and the keyword here is "think" because I'm a total noob and have no idea what I'm talking about, but I think that the Smart Context feature is now deprecated, from what I'm seeing in some of the other comments. It sucks how fast things depreciate, but on the other hand, I imagine it won't be long before SillyTavern has an easy-to-use, built-in extension to make this process easier because things in this AI world are moving so fast.
