"How to give GPT my business knowledge?" - Knowledge embedding 101

Подписаться 111 тыс.

Просмотров 168 тыс.

50% 1

A step by step guide on how to create your own knowledge base embedding, from prep knowledge data to retrieval augmented generation
🔗 Links
- Follow me on twitter: / jasonzhou1993
- Join my AI email list: www.ai-jason.com/
- My discord: / discord
- Finetune LLM video: • "okay, but I want GPT ...
- No code alternative: relevanceai.com/
- Github repo: github.com/JayZeeDesign/Knowl...
⏱️ Timestamps
0:00 What is Knowledge embedding?
4:21 Core business use cases
5:52 Step1 Prep knowledge data
6:25 Step2 Create embedding
8:34 Step3 Similarity search
9:55 Step4 Retrieval augmented generation (RAG)
12:23 Step5 Deploy
14:49 No code alternatives
👋🏻 About Me
My name is Jason Zhou, a product designer who shares interesting AI experiments & products. Email me if you need help building AI apps! ask@ai-jason.com
#gpt #autogpt #ai #artificialintelligence #tutorial #stepbystep #openai #llm #langchain #largelanguagemodels #largelanguagemodel #bestaiagent #chatgpt #embedding #openaiembeddings #wordembeddings

Наука

Опубликовано:

16 июн 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 312

@AIJasonZ 10 месяцев назад

A few people asked “why only vectorise one column instead of the whole csv?” Adding a few more explanation here: So vectorise is mainly for search, and the column to vectorise can be considered as “index” or “id” of the dataset; while the data it return will still be in question/answer pair; The reason I want to vectorise only one column is because: 1. It save cost - vectorise using embedding model which means every token we vectorise generate cost 2. It increase accuracy, in this case I want to only search for past customer email instead of sales response; search both column might return wrong answer “e.g. search for “interested in learning more”, it can return pair: “client: stop sending me emails; sales: understood, let us know if you are interested in learning more in future!” Hope this help!

@ozfish17 10 месяцев назад

It seems Embedding enriches your search query. how about answers? In your example, do you 'train' llm with Q&A pair?

@AIJasonZ 10 месяцев назад

@@ozfish17 yep, it return both Q&A pair!

@Taskade 9 месяцев назад

Jason, brilliant step-by-step guide on knowledge embedding! Your breakdown of the process was super insightful. I'm curious about how AI Agents in Langchain perform, especially in long-running scenarios. Hope you'll consider diving into that topic in the future. Keep up the stellar content!

@sandeepbansal1195 8 месяцев назад

So if you want the output response email to be generated by the LLM based on a specific tone, why wouldn't the 2nd column be a part of vectorizing the dataset?

@csss142 7 месяцев назад

Hey Jason! What would be the best way to do this with financial PDFs? I want to ask questions and get accurate insights from the large documents. Would using embeddings be best or the fine tuning from your other video? Thanks! @AIJasonZ

@psychxx7146 10 месяцев назад

Small channels like this are the ones that hold the most values.

@Helpsmallbusinesses 10 месяцев назад

In 2 minutes and 54 seconds you explained what is vectoring better than any other video online. You made it easy. Thank you!

@shivamroy1775 10 месяцев назад

Absolutely great video, I loved that you took the time to explain everything in theory and then went on to give a detailed walkthrough of the code. Please keep posting such videos !

@sidavidsin 10 месяцев назад

Thank for sharing your knowledge with us, your channel is literally a gold mine of information. Keep doing what you doing, Jason!

@_arman_ 8 месяцев назад

Man... you have a serious gift for teaching! This is super helpful. Thanks.

@fuxxs5994 10 месяцев назад

I really love your style, first explaining the theory and then demonstrating it by an example

@muhammadanasazambhatti2772 10 месяцев назад

Thank you very much! Nobody explained Embedding and Vectorization like this! Thank you again!

@humadi2001 3 месяца назад

I've watched many video on this topic and I can say that your simple examples has covered most of what I need to know. Thanks Jason.

@Optable 10 месяцев назад

You have helped the community so much with this valuable content. Keep it up my friend, i'll be watching!

@funkyboodah 3 месяца назад

man you have a really rare ability to explain super complicated things in a very simple way and organize the information so it's even more clear. Bravo and thank you

@SaminYasar_ 10 месяцев назад

Keep it up man probably one of the only channels with incredible value

@davidkwon1233 10 месяцев назад

one of the best channels out there, really appreciate your content!

@normanluismadrid422 9 месяцев назад

this is virtual gold, mad props to jason for clearly describing complex topics and even showing practical application, saved me hours of research lol, it'd be great if you can touch up on the various services out there that offer AI services that embed, and how they compare in performance, pros / cons etc.

@christhornham Месяц назад

Outstanding. Your ability to explain complicated topics is incredible. Thank you.

@wojpaw5362 7 месяцев назад

Absolutely outstanding. I liked, subscribed and shared. Best explanation of knowledge embedding I have come across!!!!

@JJ-vq8mu 9 месяцев назад

Great job and appreciate a lot on sharing your knowledge. Looking forward for Open LLM content.

@stepkurniawan 10 месяцев назад

yo bro.. i really like when you explain all the step-by-step and all relevant tools out there! thank you!

@koen.mortier_fitchen 10 месяцев назад

Thanks for your work Jason. You're one of the best, and I follow tons.

@TheDestint 9 месяцев назад

This is super duper helpful man ! Great work and thanks !

@devinoutfleet1998 7 месяцев назад

Bro... you are incredibly smart and are a great teacher. This is going to provide 10x value to my users

@aliq6709 Месяц назад

This was super helpful. Thank you, Jason!

@half_way_expert 10 месяцев назад

Another great video! Thanks Jason, keep up the excellent work

@michalf16 10 месяцев назад

Love your content good sir, tuned for all next videos you are the leader

@jasonfinance 10 месяцев назад

the best video about embedding ive seen; thank you!

@ridg2806 6 месяцев назад

Really high quality content, thank you Jason!

@PlectrumShorts 9 месяцев назад

Great tutorial! You covered a LOT of ground quickly, but thoroughly. Haha. Nice work.

@kylelau1329 10 месяцев назад

have been waiting for this video, Thank you!

@manojnaidu619 Месяц назад

Cannot be more valuable than this. Loved it 🎉

@stevi32800 9 месяцев назад

I really like your video. You knows how to reach the people attention. Please make more videos like this 😊

@pietdebeer7972 9 месяцев назад

I'm blown away. Thank you!!

@averagegamer9513 10 месяцев назад

Great video as always, Jason. Thank you for making one of the few channels with genuine AI tools video that actually demonstrate implementation and applications rather than hyping up the content through sweet talk then simply dropping an affiliate link.

@senxo.visuals 10 месяцев назад

This! I feel so grateful that the RU-vid algorithm blessed me with Jason's channel. Beautiful explanations and clear steps.

@koen.mortier_fitchen 10 месяцев назад

Yeah, he's one if the real ones. I've asked him if he could add a github for the code. It's the only thing this channel lacks imo.

@frankchangshow 7 месяцев назад

@@senxo.visualssame feelings here

@growthub8541 10 месяцев назад

So helpful! I started using relevance ai because of your videos & just as a no-code developer been able to build some sick ass LLM chains with Zapier Custom HTTP Requests. I have my development team even using it & it’s definitely speeding up our velocity to iterate🙌🔥

@AIJasonZ 10 месяцев назад

thats great to hear! 🤘

@gautamdawar5067 9 месяцев назад

This is pure gold. Thank you so much!

@robertcormia7970 5 месяцев назад

Very well done! Straightforward to follow!

@aliyousefi9735 9 месяцев назад

you're the man Jason, great content!

@nguyenvanduc2000 Месяц назад

I have the same idea in mind. I have tons of product documents that I wish I could just ask an agent something about it instead of scrolling hundreds of word pages. I really appreciate your video man.

@VaibhavShewale 10 месяцев назад

this is just awesome, now people who didnt had idea now dont only have idea but also reference

@king94596511 10 месяцев назад

The video is very inspiring and straightforward, a valuable lesson

@scratch123 8 месяцев назад

Thanks Jason this was a great tutorial! :)

@AlessaOxygen-ot4rl 5 месяцев назад

This is hilariously good. Thanks for this wonderful ressource!

@kurtcampher4716 9 месяцев назад

thank you for this As a dev with no AI experience, you really make it easy to understand

@manideepatalukdar9201 9 месяцев назад

Great video! Very simple to understand.

@verasalem5071 10 месяцев назад

Love your content, very easy to digest and understand. The only recommendation I would give is to use other embeddings and LLM models besides OpenAI. Mid/Large sized companies cannot use OpenAI in their environment because of legal issues around OpenAIs data retention policy. Alot of companies want to develop their own implementations so including other models like Llama 2, Vicuna, etc would allow you to reach a bigger audience.

@AIJasonZ 10 месяцев назад

yea great points, thanks for the recommendation! totally get that company dont want to send any data to OpenAI LOL

@Ascended23 10 месяцев назад

+1 for using more open models. I love your content and the approach you take to your videos. But even though I'm not a big company I just value using systems that are open instead of closed.

@kylearnold9647 8 месяцев назад

Thank you! This was incredibly helpful

@farid3101 3 месяца назад

I am really surprised that these tools can help so many businesses doing the low-cost and autonomous response specifically for customer service! Great video!

@ivant_true 2 месяца назад

you make really useful videos man

@MrDe0 8 месяцев назад

This is GOLD !! Thank You !

@rahuliyer6007 6 месяцев назад

Came here after the fine tune model video - looking for exactly this. Thanks!

@coldestlin 4 месяца назад

当中间向量查询的结果出来, 一下子就了解了整个流程, 非常赞. 原来是拿向量查询的结果, 再去扔给llm, 当作promt instruction, 然后让llm给出答案.

@YangYang-rh8uy 2 месяца назад

Exactly want I want , thanks Jason.

@chrisvienneau3366 10 месяцев назад

Great content and love the intros

@karankanchetty8320 3 месяца назад

Great job. You deserve more subscribers.

@NickWatching 4 месяца назад

Amazing explanations, thank you!

@markieuanroberts 7 месяцев назад

Awesome explanation, thanks.

@CyberSQUID9000 10 месяцев назад

top tier content!!!!

@arunkabilan 10 месяцев назад

Great explanation

@andrzejpec4886 10 месяцев назад

Big thank you ❤

@Artificial_Noob 10 месяцев назад

Great video man! I hope you can cover more "No Code Methods" for beginners like me that are not very technical! The last part of this video was GOLD for me. cheers!

@pietdebeer7972 9 месяцев назад

Same here

@maciejbalasinski2419 10 месяцев назад

Thanks for No coding alteratives

@tahunal 10 месяцев назад

Bro you are awesome.

@tauraik 10 месяцев назад

Amazing content my guy Amazing

@sameergamer4567 9 месяцев назад

Great video

@fenderbender2096 9 месяцев назад

Very nice video.

@slimyelow 9 месяцев назад

Excellent.

@kalifalau1455 5 месяцев назад

very helpful！

@ayusharora2019 10 месяцев назад

Amazing!!

@desiderata2745 10 месяцев назад

Thanks!

@Gingeey23 10 месяцев назад

Great video Jason, however the biggest challenge for companies will be ensuring that commercially sensitive information isn't fed into hosted LLM models due to security concerns. Would be really interested to see how you would approach this challenge, and potentially try to deploy this tool locally? keep up the good work!

@AIJasonZ 10 месяцев назад

Thanks mate! Yea I agree, I heard business talk about sensitive information a lot, especially ones with clients data; There are 2 ways I see it can be solved now: 1. Self host LLM, using Azure self host version or even using open source models; so you don’t send info to openai 2. Anonymoulyse your input/output data, so openai don’t have a clear idea that data A is from company A;

@senxo.visuals 10 месяцев назад

If using hosted LLM like OpenAI's this would probably 1. require just a lot of manual work with clearing all the data or 2. first pushing the data through lighter local LLM with a task to clear any sensitive information (like they used one LLM to create training prompts for another LLM). Just a thought, tho

@Grumptr0nix 6 месяцев назад

This is exactly what I was looking for... I have a tremendous amount of assets (Requirements docs, project plans, etc) that we've created over and over for all our engagements, and I'm trying to find a way for us to stop reinventing the wheel. All of which are in our Google Drive, but I'm having trouble conceptualizing how I'd be able to turn that into vectored data (you talk about text splitter, but I'm still a bit confused about its application). Anyways, I'll do more research but this is amazing content. Thank you.

@Grumptr0nix 6 месяцев назад

And for sure, the legal issues with our business data and OpenAI that is discussed in other comments have been a blocker for us as well, but at least there's options.

@gkennedy_aiforsocialbenefit 10 месяцев назад

Hey Jason thanks for the always excellent presentations and information. The Streamlit and RelevanceAI information were interesting and useful. Relevance reminds me of another great product, Flowise.

@frankchangshow 7 месяцев назад

I don’t know if I should use stack ai, relevance ai, or flow wise. Going into decision fatigue now

@user-ps3jj1ey5k 10 месяцев назад

解釋得非常清楚

@camach28 10 месяцев назад

It would be amazing if you could make a video creating a knowledge base using long pdfs as source,, and use gpt as well to make an expert assistant in a topic.

@frankchangshow 7 месяцев назад

Yes like if the data source is like a book and we want to search the contents in it giving relative data like “I remember this part of the book saying something like this… where was it?” … or “the book had this story … where was it and the main ideas”

@lesteroliver911 10 месяцев назад

you are just amazing :)

@KarlJuhl 10 месяцев назад

Great resources Jason, I will add to the flood of comments - you are a great communicator and you move at a good speed. Thanks for sharing! It is interesting how many langchain UI apps are being built. Relevance AI looks to be the most integrated from end to end, with such an easy deploy process. I am curious to know your thoughts on using a UI tool like flowise or relevance AI versus custom programming.

@frankchangshow 7 месяцев назад

I have the same question

@kiraakamaru 10 месяцев назад

This is exactly what I was looking for, I have a question Jason: How can we secure our company personal data?

@fab_spaceinvaders 7 месяцев назад

subscribed 👏👏

@vb7913 7 месяцев назад

Hi Jason, fantastic video! So if i understand correctly, this whole concept works purely on the quality of your examples and more importantly how your prompt is structured, as the prompt contains instructions, input and examples ?

@rverm1000 9 месяцев назад

great video! is that enough info to go out and start building a customer response ai for other people or businesses?

@ZorinsFactFrenzy 2 месяца назад

Hey @AIJasonZ, great video! I'm curious, is there a method to retrieve the confidence level from the embeddings? Since it's possible that not all the information will be present in the embeddings, it would be helpful to have a way to handle such scenarios. For instance, if certain information is missing, perhaps the system could respond with "response not found" or trigger another action like calling an API.