Can Jina handle sites with lazy load? Looking at dealership websites

Thanks for sharing, bro! Greetings from Colombia

pure gold, thanks man!

I like this format of video...background has a large monitor...Nice video

I wonder how this would handle dynamic content: as in scraping websites where you have to click stuff to reveal valuable content.

It was quite useful for me.

No, I know why you want long-term memory-because your AI girlfriend doesn't remember your past conversations. She forgets you, that's why.

Hi. I have a video request. Is there a way to contact you?

Great stuff man, thanks a lot!

For experimenting I would recommend using no database at all. You can simply use the cosine similarity (i.e. from torch functional) or quickly implement it and you are nearly done. Just use some argsort to get the best matches. It's like five lines of code or so. For easy store/load you can use pickle to serialize/unserialize the object that holds the embeddings. It is fast on CPU too, but of course you can run it on GPU without any bigger changes. No services required.

I wonder if you would update it to be able to use gpt-4o-mini as its much cheaper

Im curious how you handle pages where the content exceeds token window

what happens if we use all the 25000 cases, will it work?

you havent updated us on how much does scrapegraph-ai takes in comparison

Would be cool to make AI website scraper that strips away all javascript bloat from a webpage and converts it into lightweight basic html page while preserving functionality. Would be great as a proxy service to make loading modern web pages fast on slow phones on poor data connections. Modern web is way too bloated. I sometimes manually archive a page by deleting all javascript in notepad++ and modify image embed links to point to locally saved .png files. That takes a long time but I can reduce 5MB page down to 200kB and save that. Would be nice to have smart automated tool to do that in seconds.

Great intro and work flow. Thanks a lot.

Thank you so much for the presentation. Just in time with the latest scraping technology

Why not use Chatgpt and ask it to find information. Cant GPT just search the web itself?

I think there should be a microphone in the middle of the audience. You had an insightful discussion with them, amazing audience!

The motion-tracking is a bit distracting.

I dont know if my question is stupid, but can you tell me can we take snapshots of website and use ocr and llms to scrape the useful info, instead of sending request to that website since it would look more humanly , and also use less requests

Hi, I'm trying to scrape webdata from my Org Docs which is accessible only within VPN. Failed to goto 'docs url'. Can you help me with this ?

You should've tried Qdrant.

is it possible to run it with Ollama?

great

Nice workshop! I'll definitely try out the hybrid search. Do you recon it'll work with nomic text embeddings and ollama?

Is Openai embedding v3 model better than Bert?

Nice video sir. I have already been experimenting with the colab - sincerest thanks

Hey yall, in case you didn't get good full text search results like me, the CEO of Supabase (Paul Copplestone) sent me this to use instead: supabase.com/docs/guides/database/extensions/pgroonga

@LLMs for Devs. I'm from Jina AI. Cool that you are using our reader app. I like seeing the exact use-cases people use that one - very interesting.

Very informative! A great resource. Thanks for sharing your wealth of knowledge!!

I just use Google 🙃

very good workshop. straight to the point

This video was really helpful for the people like me looking for webscrapping tools. Though I wonder if jinaAi is really free. Is there any challenge in using it for more number of links? Does it have rate limit on hitting urls with prefix? Any clarification on this is appreciated. : )

Great video!! Thanks for sharing the code! One question though: Inside A:tier code -> "print_ai_answer" function, you wrote: for like in extracted_personality["likes"]: text_to_embed = f"The user likes {like}" current_embeddings = embedding_client.embed_query(text_to_embed) dislike_with_metadata = { "id": str(uuid.uuid4()), "values": current_embeddings, "metadata": {"type": "likes", "content": like} } embeddings.append(dislike_with_metadata) Was it not supposed to be something like "likes_with_metadata ={...}" and then "embeddings.append(likes_with_metadata)" ? I guess repeating "dislike_with_metadata" does not make a difference for the code functionality, but it was a bit confusing to understand the code for a moment. Thanks!

so what did you find out about scrapegraph ai performance , tokens

Great demo, thank you!

I don’t know how effective will this be in a long run especially due to the security update of cloudflare to block AI web scraping agents

So valueable video content! Many thanks for sharing~~

thansk, but difference or what is better gina reader or Scrapegraph-ai

Great video, thanks. Is there a way to provide our own scraped data (so we can make sure we use a good stealth scraper and get all the content), and then the LLM analyses it like this?

I guess selenium is still the choice for javascript heavy websites... any tips on this?

bro i watched 4 minutes add before jumping actual video

Thank you!!!!!!

Jina is almost perfect.. too bad it's not smart enough to scrape content from "accordions" where you first click to make the content visible. I feel a smart AI scraper should be able to grab that text and determine based on CSS class that it's probably valuable text.. just hidden at the time

"The entire internet hates him for this one simple trick"

The camera moves too much

How is your final implementation? I’m really curious about it, because make sense to have this abstracted, but we have some differences between than that can make this process tricky

such an inefficient and unreliable way to scrape the web

Please do not move the camera all the time