Yes, prompt generator is not aware of context of the whole book, which would be the next challenge. As for consistency and perfect matching to the current situation in the story it would be even harder challenge :)
this is genial and shows how to combine more ai to do stuffs and this will be the future, think to when will be possible to generate videos with consistent characters, it would generate movies from books, maybe we'll need the power of quantum computers, but we are seing were we are going.
I wonder if a top model like grok might be able to read through and generate prompts per paragraph while keeping the whole book concept in mind. and then those prompts might be commented out, so they are not read, but be processed by the image engine. "on the fly" would require a very fast llm, but if the graphic book can be pre-developed by the engines, this is a visual story book generator that could do some pretty intense stuff. like.. if the llm pre-read the book, created prompts for all the characters, stored samples of the images for each character for reference throughout the book... I think you have most of the cogs, it's just very impressive to think where this can end up in a year.
Yeah, I'm working on it :) ChatGPT4o mini was released with very cheap API, so probably going to employ it to manage concept of the book, style, consistency, plot and character development, then it will reply data to local LLM for prompt generation. It wouldn't be then fully local, but there is already max usage of GPU with current stuff, so not much can be added to current workflow if we talk about local computation. Optionally 4o mini can do also prompt generation, this way I will save about 7-8 seconds GPU work and I could use this spare power for SD and use TurboXL models with some control nets or animatediff or some picture interpolation...rabbit hole in general :) If you haven't noticed it doesn't do TTS, I load audio book that was pre generated before, so I need to also transcribe it on the fly
@@roundycreations well, your efforts are appreciated. :-) I would love to see the same story illustrated in pixel art or anime style, especially if they're european stories like the brothers grimm stuff. once that works decently, will be nice to have LLM agents code games based on the stories with the different graphic styles, leading to different gameplay concepts based on the same basic story. like.. hansel and gretel would be very different games if they were pixel style or action anime style. :-) my brain is years in the future enjoying things that might never be made. :-D
@@CrudelyMade "I would love to see the same story illustrated in pixel art or anime style, especially if they're european stories like the brothers grimm stuff." - it's just a matter of SD model used, so that shouldn't be big deal I'd say. Challenge is the consistency and context of the book translated to correct prompts each paragraph. "will be nice to have LLM agents code games based on the stories with the different graphic styles, leading to different gameplay concepts based on the same basic story" , I think we need to wait a little bit more. For now LLM can code maybe flappy birds without errors lol. But by looking at the speed of everything now, we might wake up one day and it will be there
@@roundycreations it'll be there because people like you are making the building blocks. ;-) I work in tech, I know we're years away. and it's fascinating to see early development of concepts that'll end up in much greater things. then I can say, "I used 8 inch floppy disks!" and "I remember when the guy first automated decent on the fly image generation for stories!" your efforts are also great examples of how things can work together, and these concepts can often be applied to other projects, as it's easier to see outside the box when you watch someone outside the box. :-) "one day... we'll have a box so big, the whole universe will be inside of it.. and then we'll climb out of the box."
I don't really make these blocks, people way smarter than me do those ;) but you don't need to know how lego brick is made to build a lego castle I guess
