How to scrape the web for LLM in 2024: Jina AI (Reader API), Mendable (firecrawl) and Scrapegraph-ai 

Dammit stop telling everybody about Jina my secret weapon, just stop, it's my advantage, everybody ignore it it's horrible I swear

Thank you for introducing all the latest technology for web scraping!

RU-vid algorithm is just insanely good at what it does, this exactly the content I needed and I think I have found what I want to dedicate my life to as a professional. Thank you for the video, I will buy your course as fast as I collect the money.

The reader API tip is so clutch. Thank You!

Just started wondering about web scraping and here you are. Thank you.

How did I not get your content sooner? Love it!

Came at the perfect time. Very good video. Thx 😊

Thanks for adding a new project to my to do list!

keep up the good work! - this is an awesome presentation!

Amazing video, thank you

chigga dropping bomb content, meranwhile i made a comment analyzer for highly detailed videos which have 100+ comments, and dint have time for going through all. man, sometimes you dont need to build an ironman suit to do simple shet.

🎯 Key Takeaways for quick navigation: 00:00 *🚀 Introduction to web scraping for LLMs in 2024* - Overview of startups pivoting to web scraping. - Mention of Mendable and its "fire crawl" tool for scraping the web using large language models. 02:06 *🔍 Scraping competitors' pricing pages* - The process of scraping competitors' pricing for market research. - Introduction to tools used for scraping: Jina AI, Mendable, and Scrapegraph-ai. 03:01 *🧠 Understanding "Tik token" and its application* - Explanation of tokenization and encoding in web scraping. - Discussion on the cost implications based on tokenization. 05:17 *🛠️ Setting up scrapers with Beautiful Soup and other tools* - Description of different scraping tools and their setup. - Comparisons among Beautiful Soup, Jina AI, and Mendable based on ease of use and output. 07:32 *📊 Running scrapers and analyzing outputs* - Execution of web scraping and evaluation of the output from different tools. - Analysis of readability and format of the scraped data. 09:37 *💰 Cost comparison and effectiveness of scraping tools* - Comparison of costs associated with various scraping tools. - Evaluation of which tool provides the most value for money. 12:53 *🤖 Extracting pricing information using OpenAI* - Utilization of OpenAI for extracting specific data points. - Challenges and strategies in obtaining clean and useful information. 17:20 *🌐 Overview of Scrapegraph for advanced web scraping* - Introduction to Scrapegraph as an open-source project. - Examples of complex data extraction and its accuracy. Made with HARPA AI

Underrated glad I found

Greta video! The open source tool looks great! As an aside, I use instructor and pydantic classes to get the LLMs to provide the JSON as I expect it. In my limited experience, dspy wasn't as explicit as I wanted.

Gold!

The transcript at 1:39 states that you are using large sandwich models. This must be a brand new type of model - mouth watering indeed. 😂

But the first problem that all crawls need to face is how to avoid being blocked.

I feel really sad. that you publicly talked about Jina. I used to feel special knowing very few people are aware of it lol

Jina love it...

Thank you. I'll be "away" for a while while I conquer the...I mean save the world!

16:05 Worth trying out GPT-4, I find it more accurate at following instruction.

Is the LLM community really not aware of 40 year old Natural Language Pre-processing methods developed for data mining and NLP?

lol that's awesome!

GPT 4o can do this now. Just tested and it's awesome.

Can anyone speak to the architecture or other tools to prevent detection using beautiful soup as he mentioned? What would be the best process to avoid detection and what tools I wish you elaborated there considering it’s the subject of video in large part.

If anyone’s having issues viewing the notebook on GitHub, it’s GitHub’s fault. Feel free to clone it (the cod e is there, GH just couldn’t display it recently: stackoverflow.com/questions/78501731/error-nbformat-when-uploading-to-github-from-google-colab)

What are the good and easy to use tools with langchain? Llm is not very useful without such tools, even it has no idea about the date today.

How do these tools cope with CloudFlare operating on the target site, which attempts to block scrapping?

Great presentation! I'm surprised about jin ai free scraper that doesn't require an API?!! I guess it might be shut down soon for public access

I used scrapegraph ai and was also stuck to get cost, but then I just took the cost my making some changes inside the scrapegraphai library as internally the library is using langchain and langsmith so it was calculating the cost.

youtube really know what i am looking :V with python craw a website with LLM is simple just a few line of code . back to 8 year ago i used python tool do a same thing with higher effort . right now , i m trying to mixed data from website/ database with knowledge map for observation view then i could find the short path according its , that will taking less time to read entire book in this field , just focus in some topic but still get the result . nah but you introduced the method with LLM . thanks

How well does Jina do with bigger sites with anti-bot protection?

Would these work for a dynamic website

Using jina now hehe. Does anyone know if you can get better results from amazon?

If I'm going to scrap millions of pages regularly, no way in hell AI would come anywhere close in accuracy and efficiency than a plane Http request or browser load and Jsoup parsing.

Haven’t watched it fully yet, but I’m really curious to see how it handles the looming threat of model collapse. edit: Yeah it didn’t talk about it. It’s going to be hellish when the internet becomes increasingly flooded with LLM output

Can you please fix the camera please already feeling dizzy within 60 seconds due to constant camera movement!

i tried to read or download the Web_scraping_for_LLM_in_2024.ipynb but its not readable, can you replace it ?

it works in portuguese?

What’s the best way to get in touch?

these scrapping tools are impressive... but they are not ready for scrapping full website with 100s of webpages.. unfortunately, there is still significant a room for manual scraping..

Damn, bro get ready for heavy lifting) baldness is coming Been there, you’ll look much much better!

broken link to github

That s basic stuff, I feel like it s 2023, and I was late to the party too

none of these seem better than Trafilatura?

Fix your camera thats annoying AF

"how to block these fuckin idiots AWS servers to protect your website" next

Fire crawl is to to expensive

intersting! although I was distracted by your attire... Seriously I was not born 30 years ago man, but can we dress a bit better for a presentation?!

You should definitely wear pants.