Timestamps: 01:02 PDF demo (analysis of 1000-pages of annual reports) 06:01 Visual overview of the multiple pdf chatbot architecture 17:40 Code walkthrough pt.1 25:15 Pinecone dashboard 28:30 Code walkthrough pt.2
@@johnpaulpayopay1074 Agreed. I tried to use Vicuna7B to replace the OpenAI(my machine don't run 13B). It works OK, except it can't do query on the entire PDF. Like when I ask it to summarise the whole book, it just say "I don't know" trying to find ways to fix it. Any one here know how to fix it, please let me know.
The excalidraw documents are invaluable to the presentation and really help visualize the flow of data between the pinecone database and langchain is used to make it all work. Thanks a lot for those diagrams, your efforts are not unrecognized. The possibilities of this are literally endless
1. What is the tutorial about? - The tutorial is about how to chat with multiple massive PDF documents across multiple files, specifically Tesla's 10-K Annual Reports. 2. How many pages of PDFs are involved in this tutorial? - The tutorial involves around a thousand pages of PDFs from the 2020, 2021, and 2022 annual reports of Tesla. 3. How does the chatbot work when searching for specific information in the PDFs? - The chatbot is able to cross-check and provide a reference page when answering a specific question about the risk factors or financial performance of Tesla over the past three years. 4. Can the chatbot analyze multiple PDF files simultaneously? - Yes, the chatbot is able to analyze and answer questions about multiple PDF files from different years of Tesla's annual reports. 5. Is the tutorial code available for public release? - The code for the tutorial is not currently available for public release as it is still being tested and may be buggy. 1. What is the process of converting PDF documents into number representations? - The PDF docs are converted into text, as PDF is binary and needs to be in text format. - The text is split into chunks to fit into Open AI's context window. - Open AI creates embeddings, which are number representations of the text. 2. What is a vector store? - A vector store is a database that contains number representations of documents in different categories or spaces. - The vector store can also store the text of the documents and relevant metadata. 3. How does GPT-4 retrieve information from the vector store? - The question is converted into numbers and a specific namespace or box is specified to retrieve relevant documents. - The relevant documents are combined with the question and GPT-4 looks at the context to provide a prompt response. 4. What is the challenge of analyzing information across multiple namespaces or years? - Extracting the namespace from the question is required to search for information in the relevant namespaces. - The model needs to dynamically determine which year or namespace the user is referring to. 5. How can GPT-4 assist in extracting the namespace from a question? - GPT-4 can be used to extract the namespace from the question and dynamically determine which year or namespace the user is referring to. - This allows for analyzing information across multiple years and namespaces. 1. What is the purpose of the script called "ingest data"? - The script called "ingest data" is used to load each PDF report in the reports folder into text. 2. How does the dynamic context work in the system being described? - The dynamic context specifies the namespaces to look at for relevant documents. - When there is a question, the system reverts the question to embeddings and checks the specified namespaces for relevant documents. - It retrieves the relevant documents for each namespace and then proceeds with the usual procedure. 3. What is the website called "secant Alpha" used for? - The website called "secant Alpha" is used by many investors. - It provides information on revenue growth. 4. What happened when the speaker asked the system about Tesla's estimated revenue growth for 2022? - The system searched for and found the relevant documents from the 2022 namespace. - The system calculated the estimated revenue growth for Tesla based on the consolidated statement of operations. - The result was surprising and unexpected. 5. What is the output of the "ingest data" script? - The output of the "ingest data" script is a JSON file containing information on each PDF report in the reports folder. - The file includes information on the year of the report and the name of the company. 1. What is the purpose of page numbers and references in the UI of the project? - They allow users to easily locate and reference specific pages and original sources. 2. How are the PDFs translated into text in the project? - The PDFs are ingested and split into different categories and chunks of 1000-1200 characters, with each chunk assigned a namespace. 3. What is the purpose of Pinecone in the project? - Pinecone is used as a database to store the chunks of text and metadata, converted into embeddings (vectors) for similarity calculations. 4. What are some limitations of Pinecone, and how are they overcome in the project? - Pinecone has limits on the number of vectors that can be inserted at once, so chunks are split into smaller groups (e.g. 50). The API keys, environment variable, index name, and number of dimensions must also be specified correctly. 5. What is the purpose of the dimensions in the embeddings? - The dimensions represent different spots in an array of vectors, with each dimension containing a number representing a specific aspect of the text or metadata being analyzed. In this project, OpenAI creates 1536 dimensions for each embedding. 1. What has been trending on GitHub for the past couple of days? - Answer: The transcript does not provide a clear answer to this question. 2. What are the indexes and namespaces in the code? - Answer: The indexes and namespaces are components used in the code to retrieve information from different sources. 3. How can one explore what the vectors look like in the code? - Answer: One can click "fetch" to see what the vectors look like, which includes the namespace, ID, and decimal number representations of the text. 4. What should one do to ensure successful ingestion in the code? - Answer: One should ensure that their config has matching namespaces as in the dashboard, set environment variables correctly, and avoid tampering with versions to avoid breaking changes. 5. What is the purpose of the second phase in the code? - Answer: The purpose of the second phase is to chat and retrieve information dynamically by specifying the namespace to retrieve information from. 1. What is the custom QA chain and what does it do? - The custom QA chain is a tool created by the speaker. - It takes the model, index, and namespace to effectively set the stage for the Standalone question and retrieve specific name spaces and relevant documents to provide a response. - The custom QA chain is just one of three chains available in Lang chain, which includes chat Vector DBQ a chain and Vector DBQ a chain. 2. What is the purpose of the chat history in the code? - The chat history is set to nothing and is not directly relevant to the custom QA chain. - It is used in the API implementation of the same logic seen in main.ts. 3. How many files were tested with the custom QA chain, and what were the results? - The speaker was able to read three different files with over close to a thousand pages of in-depth financial analysis. - Gpt4 was able to analyze all three years and provide decent analysis. 4. What is the purpose of the front end in the repo? - The front end in the repo is already available for use. - It is an adaptation of the custom QA chain tool. - The speaker is experimenting with talking across different PDFs. 5. What is the compound annual growth rate for Revenue over the past three years? - The transcript does not provide an answer to this question. 2. What kind of questions were asked about Tesla's revenue? - The questions asked were about growth potential, profitability, and risk factors management. 3. What is the K1 ? - K1 is the number of reference documents that are returned per PDF when using GPT to analyze data. 4. What kind of future changes will the repository make? - The repository will add features for analyzing multiple PDF files in the future. 5. Where can you find more information about GPT and its applications? - The speaker suggests checking out their workshops and signing up for the waitlist in the description section for more in-depth step-by-step details.
Thank you for your great videos, I really like them. One UX improvement comment: Try serving the PDFs as static content with your web application, then when you append #page=42 to the PDF's link, a user can go directly to that page in their browser. Very easy to implement and your users will save a copy and paste and 3 clicks.
Cool stuff, It is just unbelievable how much depth and breadth all the computing field has become, Too stretched out and too spread, Too much subjects for to explore, Chating with books is a cool idea that could really help us access and organize scientific human knowledge in all fields.
Thanks for this, your vids are some of the best on YT now for this type of thing as right now in this domain there is only two words for all the content out there - information overload!!! I am still going through your prev video on PDF ingestion and processing, thank you for sharing your invaluable information in a style that makes it understandable.
I work in an academic research setting researching an extremely niche subject in material science and these kind of tools are what I have been waiting for. I can't wait for us to be able to upload the papers and books that contain all current available knowledge on the topic and be able to interact with that knowledge pool.
Can you please show how the front end connects and how to build it? This is definitely the missing piece of the puzzle for me. Thanks for the videos. They're great!
Great video. But the problem with your solution is you have hit the OpenAI api at least 5 times and it would make it costly and not scalable. Other than that, a good project.
Just found this channel, 16k subs now, not for long. You are making great tutorials that I will say I should have made months ago. Keep making videos you can hit 2m subs in less than a year if you make content more relevant and useful to a wider audience and more entertaining
This is the best tutorial I've seen on embeddings search, the applications of this are endless, really excited to start building. Thank you so much for the work that you're doing :)
Hey, the tutorial is one of teh best of its king in this segment. Can please share the repo link to this multi pdf one. Its very useful for others to work on.
Brother...! This is what I am talking about! My sellers are looking for this kind of capability for research as well as for searching our internal content management system.
- Convert large PDFs into text for processing (timestamp: 6:36). - Split text into chunks compatible with OpenAI's context window (timestamp: 6:59). - Create embeddings from chunks and store them in a vector database (timestamp: 7:14). - Query the database dynamically using namespaces to extract information from specific years (timestamp: 11:03). - Use GPT-4 to parse and understand complex financial questions across multiple documents (timestamp: 17:19). - Leverage Langchain and Pinecone services to structure and query data efficiently (timestamp: 18:24).
Thanks for the detailed presentation and explanation of your concept! This is really exciting to learn - subscription is set, I'm looking forward to more videos. Thanks mate!
You don't, the embeddings handle getting that crucial context information and then you pipe just the relevant info to ChatGPT to help inform the answer. It's still not cheap by any means, but much cheaper than sending all of that data. We're doing something similar at work, and it's still super cheap.
I don't think this is the right approach. It doesn't know enough context on the whole document. When you hit a prompt, he will match a chunk to your prompt, but what if the answer is contained in the context of let say 200 pages ? You could make something clever. You could aggregate chunk togethers asking the LLM to summarize and group them in "meta chunks", you could repeat the process until all years are contained into a single max limit tokens batch. Then, with the meta Data, you'll be able to perform a much more powerful search over the corpus, providing much more context to your LLM with different level of aggregation.
Something like that already exists and is called a “recursive summarizer”. The hard part is figuring out how to actually perform search over the recursively summarized documents without invoking the GPT api an unreasonable amount of times except during preprocessing. I am currently doing research in the area of passage retrieval and would love to know your thoughts on this and possibly experiment with some of your ideas.
@@user-jc3ys1yc2n Well, I had this idea while having my morning coffee, but I haven't conducted any deep research to support it. However, as you mentioned, there may be potential issues that arise. Have you considered using GPT4 to help solve this problem? Personally, I anticipate having to address this issue myself because I plan to summarize big HR data.
you lose more and more information by recursive summarization. better to use embeddings, so the LLM knows the whole context of your documents, no matter how many...
@@JanBadertscher No, of course, it doesn't have complete knowledge, that's the main issue. It only has partial knowledge based on a vector match. This is why it provides inadequate responses when the prompt is not very specific. It is incapable of extracting critical information that comes from reading the entire context. A human expert is currently better to do this.
You can control the `k` returned source documents up to 1000 if you want to via Pinecone + gpt-4 context is 8k, so you can cover a lot of context across the doc.
Your Excali diagrams are invaluable too. I’ve been trying to get into the habit of creating similar diagrams. Do you usually come up with them before you code, make them manually as you go, or put it together once you’re done?
First off, very well explained, great detail in the video - thank you for taking the time to put this togeather. I was wondering, for use-cases where the documents you are working with should remain private... is there any way to create the embeddings with some locally run model, rather than sending the raw document text in chunks to OpenAI?
Why not use Embeddings as the documents title so that instead using Gpt to do a search of the exact name space, it could do a semantic search of name spaces similar to the initial prompt, thus finding things relevant more effectively. Thoughts?
I'm extremely interested in AI and Language Models, but I have a limited understanding of where to start. I was wondering if you'd be able to recommend any programs, courses, or tutorials for those in a similar position to me. Thank you, and what an interesting video this was!
If you have a basic understanding of multivariable calculus, probstats and linear algebra; see few videos on ANN, then RNN, then transformers and BERT, and then finally LLMs. Its a long way but this way you would be able to understand it more deeply and you would be able to understand the recent developments in LLMs more clearly.
When will you release the code from this walk through (not just the "base template", but the actual code here that uses namespaces)? Also, do you know of any way to prioritize what source it uses? For example, if we have 3 PDFs for 2021, 2022, and 2023, and each declares a new color of the year, and I simply ask "what is the color of the year?" how can I ensure it gives me the most recent answer?
Great video :) What do you think about using metadata filtering instead of separating data by namespace? By giving users the ability to select metadata filters, you can provide a personalized search experience with precise filters for two or more metadata types, without the need for feature extraction from text. Keep up the good work!
have you tried this? i have tried it in python. at least I've tried metadata filtering with the new SelfQueryRetriever, and it did not work. There is a Structured Query component using comparators that does not allow it to work without some extension of the class and modifcation. Let me know if interested in hashing this out together.
@Chat with data , do you mind sharing the meta data version of the code? where you chunk and tag the documents by year and page number. I am really interested in what you showed in the demo. Impressive work!!!
hi, let try adobe pdf api , you can improve a lot your embeddings with that , the api auto detaching paragraph , title , table and image , and you can superpower your stuff , also the chunk in not more no sens text but is p or H1 ecc that have sense and you can alt give weight to data , for example h1 is more important of a end page little paragraph
So when you do Embeddings - maybe I missed it - I assume that you do not generate embedding of the entire PDF page (that you converted to text before that). So how do you chunk the text in the page? By paragraphs? By sentences? (The former would be a bit of a problem if you removed all " " while extracting PDF text.).
🎯 Key Takeaways for quick navigation: 00:00 📑 *This video discusses how to analyze multiple massive PDF documents, like Tesla's annual reports for different years, using AI.* 02:36 🧠 *The video demonstrates asking analytical questions about multiple PDFs simultaneously and extracting insights from them.* 03:21 💼 *It showcases asking technical questions and comparing data from different years in PDF reports.* 06:36 📊 *The video explains the architecture, including converting PDFs to text, splitting into chunks, and using PineCon for storage.* 11:03 📈 *It discusses how to dynamically extract namespaces from questions to query specific PDFs and retrieve relevant information across multiple years.* 35:52 🤖 *The video demonstrates the process of extracting namespaces and mapping them to specific years in PDF documents, enabling targeted queries across multiple files.* 37:58 📊 *The presenter showcases the ability to analyze compound annual growth rates across three years of financial reports using GPT-4.* 39:49 💰 *GPT-4 accurately responds to questions about growth potential and profitability based on the past three years' annual reports.* 41:03 📈 *The video outlines the architecture for using GPT-4 in applications involving multiple PDF files and suggests adjusting parameters for improved accuracy.* 42:36 🧰 *The presenter encourages viewers to re-watch the video for a better understanding and mentions upcoming workshops for more detailed guidance.* Made with HARPA AI
Thank you for this excellent contribution! It has really simplified this process for me greatly. One question: The first step (well, second step) in Phase II of your flowchart is to combine the Chat History with New Question. The next step is to submit this to an LLM to get a new standalone question. Why do you do add this step instead of immediately vectorizing the Chat History with the New Question. What does this "new standalone question" look like? If this is explained in this video, or another, could you please include the url? I'm kind of new to this. Thanks!
Sorry, I must have missed something. I've watched both videos, and I still don't know how to take a pdf and insert it into Pinecone. Do I upload it somewhere? Do I share a pdf link with Pinecone? How did you get Pinecone to interact with the Tesla pdf? How does Pinecone know that this pdf exists?
Hi there, what an awesome video. Will be trying this out this week, asap. I am wondering how long have you been coding, and more specifically how long have you been working with technology like in this video?
Sorry im new to this channel looks amazing. How are you getting that chatbot? where can i find it? Amazing channel and information you are providing though
Just as a suggestion, Put a disclaimer. I tried running something similar (tensor flow Universal encoder decoder 4 running locally + some custom RNN + SinDy)with Kongsberg groups annual statement which consists many smaller companies such as Rolls Maritime and Security. Although publicly available. I swear to everything holy, All my files vanished. No updates or crash nothing. I went to prepare some food, came back and it was like I never started the project. Hell.. even my Bills for openAI were cleared. Like I never wrote a word. Im still doubting my sanity. But the paperwork is there so I couldn't have "tripped".
Very nice video, can we use metadata filter instead of namespace , will it work? Also could you please share me your main.ts , fewshotprompt template..
I want to use a tool where I can upload 2/3 pdf files so I can ask it questions during an exam for example so it can answer my questions according to the pdf documents I have uploaded. How can I arrange that ?
Question: instead of PDF, could you point it to the database to do the same? How would one accomplish that? Thinking of a BI tool or an ERP. I think digitizing PDFs will be great for organizations that have technical machinery - example field crew sales - if they need to answer the technical question, it can be input (via voice) into GPT and reference company information. One would need to make sure it is secure (ie ChatGPT cannot store what it reads).
Hi, great tutorial! I was wondering if it were able to read the financial reports such as the net income, debt etc.. within the balance sheet or income statement?
@chatwithdata thanks for sharing this demo. In my line of work we sometimes deal with sensitive and non-public data. What risk factors are associated with analyzing sensitive data using GPT apis?
Hi Mayo, is there a way to include paramaters in the conversationchain for maximum length, frequency penalty & presence penalty? Apologies if it's simple but new to this. With your instructions + video, i was able ingest my own PDF's. It's a great start & now i'm looking to optimise the outputs.
Thanks for the great video, I'm using it very well. When I use the code you posted on GitHub based on pinecone, is there any limit to the number of PDF files/total capacity/total number of sides, etc.? It worked fine when the total size of multiple PDF files was around 84MB, but when it went up to 128MB, I got a SocketError:other side closed error in npm run ingest, so adding a PDF file to an already existing vector store doesn't seem to work well. Do you think I need to adjust the timeout for undici to be a few times longer?
This video and channel are awesome! I'm working on a version of this demo in python, would you mind adding an Apache or MIT license to the gpt4-pdf-chatbot-langchain repo so I can share without worrying about any weird copyright issues with github? (Accidentally posted this on the wrong video earlier...)
Interesting that there is a grammatical error in both the question and resulting answers. Does this mean GPT-4 is simply parroting back to the questioner OR is there an inherent grammatical error bias in the model? I am referring to the use of the plural for the management of a company and the company itself. Whereas in fact, a company (Tesla) is a single legal personality and should be referred to in the singular. 'Its' not 'their', 'it' not 'they', 'has' not 'have' etc...
