Hi, yes, that's a known issue, it's not perfect. Pdf readers often struggle with table data even in the best RAG solutions, and of course cannot easily understand any information that's not in text format already. It's mostly for poc level example, to make it work better, much more work is needed. What can be done? - check which part is not parsed properly - you can isolate the code that parses pdf and play with it - there are typically adjustments you can make to better meet your needs, but as mentioned, it can mostly handle text chapters and titles, table data and images need some other approach overall - possibility to see the library issues list, see if anyone else is struggling with same issues - possibility to check to see if there's a better library available for your needs - there typically are options. This was just one that was fast to get working. So, pdf documents are not the easiest usecase to feed for RAG, something that would be better structured is typically going to give better results. But it's a nice starting place. I'm a bit concerned myself about losing the graphs, images, and potentially having table data cut in halves etc, those can alter the results vastly. If you can find any better means of converting the pdf to any structured format that would retain what you need, that would improve the results.

Hahaa yeah, I can relate to that :)

Perfect! It's gonna be slowwww... But fully local too :)

Thank you!

Thank you for your feedback! Much appreciated!

Indeed. Seems it's not going to make it into future releases.

Thank you! Much appreciated! More to follow on the topic. Stay tuned. :)

Thank you! Very good ideas!

Well, I use both. VSCode is awesome for dealing with multiple languages and config files, very flexible. Jupyter is great when dealing with mostly Python and data files, soecialized in those. I've got Jupyter plugin in VSCode too, so I use both, depending on what I'm working on.

@@DevXplaining Fair enough... does Lab do anything vsc doesn't?

No, the preview features are always behind the preview flag. Of course the idea is that eventually they get out of preview and become just normal features and thus flag is no longer neede. (Or not, they may never become features, or may change a lot before becoming features). So anything behind the flag is more like a cutting edge sneak peak than something actually released and stable. So good for playing and feedback, but not so reliable for production use - yet. Of course you can hide any flags you need in some kind of startup script etc, so nobody needs to actually see them. Java typically uses all kinds of parameterization anyways.

@@DevXplaining yeah got it thanks. Hope they’ll make those features normal soon ✌🏼

Oh yes, you've hit one of the weaknesses/limitations of the current RAG model in general (and vector databases). They are suprisingly awesome at digesting and using huge amounts of unstructured data. But the simple examples do not even attempt to deal with bits that require specific interpretations. PDF readers are good at grabbing document data, titles and paragraphs, to make sense of it. But they currently already struggle with tabular data, diagrams, charts, and yes, math formulas. Furthermore, those are fed into vector database as text typically, and similarity search also works text-based. And when you retrieve it, you retrieve text. I'm afraid there's not much to do about it, except you can of course create a specific pre-parser for the documents, or tune your pdf processor or write your own, to decide the rules how the pdf or html is interpreted and stored. I don't know your exact use case, but my advice would be: yes, it's not working, because it's not a trivial problem to solve. Yes, you can probably solve it, but you need to rethink it. Math formulas typically benefit from more structured content. You can always handle them outside the system, perhaps store some links or references in the rag solution to find and fetch them etc. I have to say haven't had to solve this issue yet but already the tables are a common challenge and question with current gen RAG solutions. It's a decent start but more is needed.

Oh, forgot one thing: multimodal models that are able to make sense of images might be interesting to explore for formulas, to work around text conversion.

Awesome idea! Working on something else right now, but this sounds like a good plan for a long form video, and a fun one.

Give me your job, Ill do it for easy

Oh yeah, a very good point. Job market is a bit more challenging right now, so anything concrete to be able to show might just give you that edge. In that sense, even the lightweight free certificates you can do within few hours work. Probably not going to be the main decision factor, but just a little bit of extra on top.

Thank you! I did not include frontend because on top of API that is quite straightforward, and tried to keep the video compact. But I've been thinking of doing a bigger video with full stack, because it looks a lot nicer. But not there yet, at least. It would really be a trivial NextJs (or any other framework) web app that you dump into spring boot 'public' folder under sourcecode. So I'll definitely take that under consideration, but probably not going to happen soon.

Interesting idea, haven't done that yet so need to look into it, if I find something cool I'll definitely be back. Not sure how the API works for them yet, but can see how that could be useful.

Haha, should actually be delivered as sealed physical scrolls I think :)

Thank you! Much appreciated!

Hmm, perhaps a topic for another video. Short answer is... depends.

Thank you for the feedback! Much appreciated! Love to learn and share these things, I understand best by building and tinkering. Some more cool RAG stuff coming up soon! :)

Sure, I've done few in the past on my channel, but of course might be nice to redo since both Java and Spring have gone a long way since then. I've mostly been showing Spring Boot + Web module, to create RESTful APIs, and what you want to do behind that is of course up to you, but most often it includes database. But, while waiting for new video, these should still work to give you the big picture: - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-xxRw0XIoKvA.html - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KsWg6nPVwsg.html - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Sg1b0sugbtg.html Thanks for tuning to my channel!

Thank you!

Hmm ,interesting question. If I got it right, some customers would get answers based on the document, but others would get answers based on some APIs? If so, in both cases you want to inject some context in system level instruction, either from vector database/document, or an API - so that part remains the same. So when you need to answer an API, you would call the API, get the context for the question, then include that in system direction. No need for vector database, as long as what the API returns a) has all the context info needed and b) is relevant and c) is not too massive to chat in the available tokens. So in both cases, grab the context information, feed in the system level instruction like in my code. Decide on top of that which context is used for which user. One thing I'm not doing in this video, is keeping a conversational memory: I'm just asking a question, and activating context + chatgpt to handle it, but then forgetting all the conversation. So with conversational memory, would need to be a bit more careful, because also the conversational context is typically injected into the calls. Also I think would have to be careful with size, of course depending on your model token limits. And you would need to keep the contexts separate so each user would see their own personal context only (instead of same shared one for everyone). I hope this helps and I got the question right!

Thank you! It seems RAG usecases are in demand right now, and there's also uses for own internal tools. Some of these of course will appear as part of available tooling, but being able to do it yourself is great for like-minded tinkerers that like to understand things by building them, I think.

Noted, need to make a longer video then but would be fun :)

Haha, I actually aporeciate feedback on my pronounciation, lets me work on it. Thank you!

Thank you! Appreciated! :)

Haha, you know, I coded myself an IDE too, when starting to learn Java Swing. It never got full featured enough, but it was fun to build it from scratch and then use it to develop it further. Also made me appreciate people who work on modern IDEs a lot more - there's a boatload of details to solve and take care of to make them work beautifully.

@@DevXplaining i tried to do it with imgui, and failed bc macos is stupid :( and i never even figured out how to make it so JNI executes on the main thread. Altho if java adds a feature like this then ill probably try doing it all over again

Thank you! :)

Yes, a good question. Short answer is that yes, typically that's what happens, and typical software project inside any web framework pretty much ignores you needing to tap into parallelism - you just think about it when accessing background resources. But two reasons to still keep up with Java parallelism features: 1) Well sometimes you do build something from scratch, that would mainly happen if you want to do some intensive calculations without web servers, or if you are building a server. 2) Virtual Threads are currently finding their way into popular servers and frameworks, so mostly you can just enjoy the benefits, but it's at least good to get the idea why they may bring more performance when your favorite framework start using them. So mostly I think this is the message that Java platform is still very much alive and kicking, and still getting better. Spring Framework is already embracing them, for example: spring.io/blog/2022/10/11/embracing-virtual-threads Might be a fun topic for a future video.

okay great. Maybe you might consider making a video in the future for this question or you could please share your ideas here. I'm a bit confused if Virtual Threads are going to replace Reactive Programming. If they would, why? and if they won't, why?@@DevXplaining

Yeah, it's a good question. AI takes many forms, so rule of thumb is: anything you would not like to/be permitted to publish on front page of a popular magazine tomorrow, you should not send to AI - unless you know how the data is handled, stored, etc. And additionally, is your input used to train the models (which could cause it to pop up as an answer for future users). So for example using ChatGPT like I do here, I'd limit it to general advice. But for example Github Copilot Enterprise is my tool for daily work, and there are some rules in place to stop my code from leaking. So either know how your AI handles input data, or be safe and just use it for general questions. But as always, keep your secrets secret! :)

very insightful, Thank you@@DevXplaining

Yes, agreed. I was teaching Java for a decade and that getting started part for new coders is tough with it. Once you push through it does get better. So I'm very happy to see they finally added beginner-friendly features, even though old hands really don't need all that :) But much easier to learn it, teach it, do something small just for fun.

Hi, that's a good question. That's built-in the speech recognition library, so it stops to interpret the speech when there's long enough pause. So I'm kinda giving it the whole text by not pausing before I'm at the end. Similarly, the SR library also starts interpreting speech to text only when some audio begins. For any SR library (I'm using Google API in this example), there are typically settings to adjust how long pause means speech has ended, and it's time to interpret it. Alternatively, there's also possibility to introduce walkie-talkie-like push button to speak mechanism to signal end. Finally, one could also of course just interpret constant length blocks constantly, but then that might not make sense to be sent for AI to interpret.

To give a more concrete answer, here are the parameters that could be passed in my code to adjust how long pause means end of block of speech (in seconds): recognizer = sr.Recognizer(pause_threshold=0.5) # or other value

Hi, you know, that is a possibility I've been thinking about. :) Two things on the way: It takes some effort, so I need to find the time for it. Second, there's been a lot of options available currently, so need to choose which one is good. If I can find the time, what I've been thinking about would be to get something running in Raspberry Pi/other low-end hardware that's easily available. I would probably use a suitable Mistral model right now. But no promises, time is hard to come by these days :)

Thank you! Much appreciated!

Oh thank you, much appreciated! Feedback like this definitely inspires me to create more. Yeah, two problems with Oura data is that it's not real-time for performance reasons, and granularity for some datapoints is very sparse, sleep scores being another tough area because you only get them after, and some of them accumulate through the night. That's kind of by design, to conserve energy. I don't remember the granularity of SPO2/temperature, but I don't have any special tricks to overcome. I tend to treat the data based on what granularity/latency I get, aka: Typically just zoom out and get a wider view. On a sidenote, I am suprised that my Oura 3 is still alive and kicking well, previous one died early for battery issues, this one is up and running. For inspiration, here's my view for Oura data. If you just want to analyze it, Oura software and dashboard already does pretty good job in calculating the scores and showing the history. What was interesting for me, was to figure out how to combine that data with other sources, for example compare my heart rate/heart rate recovery against my daily or weekly schedule, my sleep schedule, music I am listening, etc.

Thank you! Much appreciated!

I like that Java keeps updating, outdated features are deprecated, then removed. New features are brought in constantly. Using Java 21 and beyond feels so fresh.

@@DevXplaining yeah, coding in java 1.4 vs Java 21 feels like almost totally different languages. I am happy now I can make simple program/scripts in Java and throw away python almost entirely. Looking how to use GraalVM to use Java to automatize small Scripting fit tasks without the overhead of the JVM. The best time to be a Java coder is now.

Hi, happy to hear you liked this. This is one of my favourite pet projects, and every now and then I actually turn it on and have a chat, because it's fun :) If you mean latest Python versions, sure, I typically use latest available when recording and this is quite recent. The OpenAI APIs have changed a bit, this video and code uses the latest version of those as well. So should be quite up to date. Of course few months have passed so things may have updated a bit, but as far as I know nothing that would have an impact here. I used then latest available preview model. For the coding from beginning to end.. If you take a look at the code, it's only 69 rows, so not much to it, mainly some decisions on which libraries to choose to get the results I wanted. I try to keep my videos short and sweet when I can, that's why I just shared and showed the result (link to git in the description section). Here, libraries do all the heavy lifting. But this is a fun topic so would probably not mind doing something from scratch, with all the steps. But wouldn't want to do exactly the same thing, so it would have to wait until I get some nice update idea for this. I have some plans cooking but nothing concrete yet. But I would say: It's a nice tool, simple bit of code to get started, inspired, and make it your own, take it further.

Hey, thanks for your detailed reply. That was very helpful. I’m keen to see what you come up with. I’m after my own self learning AI assistant that could do a multiple amount of things for me. Problem is, most coding includes so much limitations on the extent of what an AI platform can do. A true AI assistant should be limitless. The possibilities would be endless.

How can I DM you?

Ah, that would be a sidestep from the original theme, don't plan on that video currently. But. There are several ways to do it. I play a lot with Jupyter Notebooks for visualization, so I see two parts here. Step 1: Get the data I gather above in a nice regular numeric format, csv files tend to be nice. Step 2: Get a visualization library of your own preference, for example Matplotlib for Python, Plotly, etc, and plot that data out. Easier path could be to just do it in Excel. Once you get the data in table format, anything is easy. So that specific thing is probably not the next video on my channel, but hard part is getting the data, then many things become possible. Thanks for watching the video!

And thank YOU! :)

Thank you! Much appreciated!

Thank you for your feedback, much appreciated! Interesting finding. I've never actually compared the oura app stats to the excel exports or API-returned results. It's of course possible that they wouldn't match, but I find it a bit odd. One thing I realized playing with the data is that you easily get a one-off error, for two reasons: 1) data you can export/pull from API is of course not real-time. For data to be there, the ring must have send data to app, and app must have sent it to cloud. This makes the most recent data almost always incomplete. So data gets more reliable after it's few days back in the history.Which takes me to second point 2) there's some data that's available as it's created, but sleep data accumulates over night, and any metrics are calculated and ready for the night afterwards. so if I grab todays data, I get previous nights sleep data along with current dates activity data, this easily confuses the results. So I'd have two questions to narrow down the issue: 1) compare some data at least few days back in history. If the sleep number from API seems to not match what the app tells you, check the days around it, does it match one of them? If so, there's probably logical one-off issue you can compensate to put the data to correct day 2) if there are mismatches, are they regular so that all days are off, or are most days correct, with one day off here and there. If latter, then it sounds like issue with Oura data, otherwise I'd double check my own process and logic. Videos here - no guarantee they are more reliable source of information than Oura documentation. There's been rare cases of me being wrong as well (typically once in a decade or so :) - but Oura is also very secretive about their algorithms, because it's competitive advantage, so I've kind of reverse engineered some details, but no way to check things 100% correct. Just my observations here, sharing my path in hope to get others also interested and dig into it. But I am a big fan of gathering and analyzing metrics, treating myself as an IoT device, so Oura definitely fits that bill. :)

It sure is! Not something I have experience on, nor videos, but looks like fun as well. Here's a good writeup of how they work together. julia.quantecon.org/getting_started_julia/getting_started.html#running-jupyterlab