Industrial-scale Web Scraping with AI & Proxy Networks 

❤

This man selling wood and iron to shovel makers

@beyondfireship ad link doesnt work

​@@anze worked for me. Try now.

The pricing page says 20$/GB. I checked how big the pricing page was it loaded 4MB, so then it costs 20$ for 250 pages? That seems very expensive. Or how should i calculate the price?

Industrial scale!!!

scraping aint cheap, but theres many ways to make it much cheaper

​@@arteuspw what do you mean by 1gb/$1? You mean browsing 1gb of data for a dollar with a single proxy? How many proxies do you get for $1?

@@arteuspw Please tell me where to buy them at this price.

Is 20$ per GB considered expensive? I wonder how much could you scrape from a site like amazon for that GB... surely a lot ?

interesting - 8 years of freelancing and never had one such request 😮

And with a freelancer, the business has the advantage that YOU break the terms and conditions of the companies you scrape (are legally liable and suable). Not the business 😊 so a cheap code monkey and legal scape goat all in one 💪

Where do you get these projects from?

@@dinoscheidt Wait can they really do that? They are the ones who wanted to scrape the data in the first place.

I'm not even a freelancer and I cant count the number of times on two hands Ive been asked to make someone a website. They think because I'm a web developer I am just some guy who goes around making websites willy nilly. And the few times I have actually went through with helping someone out, they want everything Wix or Wordpress provides and have the audacity to suggest I shouldn't be asking so much in pay when a drag and drop builder can suffice.. THEN USE THE BUILDER GOD DAMMIT. My knowledge is wasted on front end work anyways.

Please do!

Dont be shy :p

so that's why I copy full selector of the element and work with it in puppeteer.

AFAIK puppeteer doesn't support finding elements by xpath, so what do guys use?

@@MrNsaysHi well, real men write their own html parser and query language. But peasants like myself use css selectors with document.querySelectorAll.

You're right, wanted to said that... But don't have money to spend on the browser. Is there an alternative?

​@@yvanguemkam4739 you can host puppeteer yourself and pay for a proxy service if you need it, might come out cheaper (but more work obviously)

Scraping isn't illegal

and individuals

@@dislike__button if it's against Terms of Service of the given site, it is

@@aresakmalcus6578 I'm not a lawyer, but how can it possibly be illegal? It can be against ToS, sure, then the website owners can surely act accordingly, i.e. ban your account on the said website, ban your IP address, and so on. But illegal? Are there any laws out there that prohibit collecting public data? Are there any cases of people getting sued for scraping? I haven't heard of such, maybe you can provide some examples. Also, there are 8-figure businesses built on scraping, like Ahrefs or Semrush.

@@Andrew-zy7jz Exactly! Very good example.

same, gives me shitton of satisfaction

thanks Jesus

Unfortunately GPT4 is still too expensive to use in projects and gpt3.5 is still too stupid.

@@alejandroarango8227 it's stupid enough so you still do much of the work yourself cause eventually it's just a tool to help and personally it helps me enough

@@0187 exactly what i feel

He already has five prototypes of Neuralink chips plugged into his brain, linked to the Web via 5G, and he is using digital clones of himself (coded in JS of course) to make more video content (with the help of ChatGPT). That makes him the most powerful being on the planet. Praise the Cyber-Jeff! 👾

With ChatGPT of course

Mmm

Downloading CSV files would typically not be considered _"scraping"._ You don't have to scrape the data out of a CSV file -- it's already data.

@@DemPilafian you just wanna falsify my statement but scraping for csv file is as valid as scraping for pdf files. I specifically wanted to scrape soccer analytics websites for those csv files. Hope that puts it into perspective for you .

how much did u end up spending

He has got a website offering courses. I bought the Angular one myself and was really good.

could you give some tips to not getting ip banned?

@@donirahmatiana8675 puppeteer-extra library and the puppeteer-extra-stealth plugin. If that doesnt work, you'd need rotating proxy like that of bright data as mentioned in the video.

@@nichtolarchotolok are you not using scrapy? I always thought of scrapy as the most convenient solution.

@@jacekpaczos3012 I started off on the nodejs route and havent had the need to try the python way of doing this. I do remember trying scrapy in my early days but for some reason puppeteer felt more intuitive to me. That is probably because I felt more comfortable writing javascript code.

there's Pyppeteer

@BeBop No, it's Pyppeteer

@@bebop355 *pyppeteer tho

did this materialize?

​@@JGBretonlmao 😂 asking myself the same thing

what are the others that are better?

@@sciencenerd8326 the company has made some cheap proxies using the machines of AWS for examples (they don't have many IPs but they do the job for many websites). And I think there are cheaper services like ProxyRack.

​@@hamza-325I have a task of selenium bot I have 1000 account but need 1 ip for each account to make request to the website and do the work any idea or paid service for that

I avoid using it to the maximum, it is a waste of server resources.

You could just close the browser and open a new one every time you use it to avoid memory leaks

Well thought!

smells like ad

@@klapaucius515 Do you mean that for my comment or the video?

ikr, then you can just pay brightdata $10,000 and go on to make $52 for the data you've scraped.

theres many videos on Fireship where he jokes about living dangerously and letting the cred be seen 😂 obv temp stuff

​@@cymaked or just F12 to let somebody waste their time

Hi I'm willing to help# I'm a dev looking for commission

I mean that's way more readable than 1200000, this is a pretty common practice

May god bless you

BeautifulSoup is better anyways :P

@@danielsan901998 or Pyppeteer

@@danielsan901998 Correct, that's not an even comparison. However: BeautifulSoup >> Cheerio

It's a proof of concept/tutorial, not an explicit recommendation for bulletproof boilerplate. Context, eh? :)

Maybe outputting the html every time to openai then having that pick the query selector then insert into the script. Do have to be very specific with your prompt because it often replies with: The query selector is: a.carousel

bright data is insanely expensive so that's the protection against DDoS lol. You'll run out of money before you even have the chance to send enough traffic to cause a problem

Yea, I think classes should be autogenerated, at least after every deployment if not every request. Fast and dirty solution would be to use openapi sdk to prompt ChatGPT to generate document query code and eval it

@@makkusu3866 You can select elements based on attributes or lack of attributes, or you can use pseudo-classes such as :nth-of-type. There are dozens of them.

@@trappedcat3615 yea, that is how i wrote scraping for other website, i targeted div elements with style X which often ... doesn't change cuz... why ;D

Most websites with random ids/classes names still have a common and repetitive structure. Axios + Regex and you'll process ~10 time as much pages as puppeteer, with minimal bandwidth by default and simpler code. Just validate the output with a strict schema (as you always should) and you'll maybe have to update it once a year at most. Puppeteer's only real advantage is TLS fingerprint

​@@arthurchazal3064 you can also use sg like cheerio as a middleground between an entire headless browser, and parsing html with fukken regex (chad move tho ngl)