Hello, Adrian! Can I ask you a stupid question? I'm only beginner developer and looking for your course. Why you need to build a custom web scraper from scratch, if there are a lot of services and desktop apps? Yes, they are paid, but they also provide their IP and proxies included, etc. I'm just curious, what are the benefits in building your web scraper from scratch instead using ready to go services?
Does your course explain how to input data as well? I want to have a flow like Enter Text Field --> Click Button --> Click Selector Button --> Click Download Button --> Save DL in Database ; Repeat Process but with different info
Is this a standalone js file that can be run if all dependencies are installed? I didn't see where you included the openai api if it was needed. Thanks.
Hi Adrian, love your content. Saw your video about getting cookies and linkeding profile scraper. If I got you right I could get the scraper running on a server but should use a residential proxy simulating my location. Does this need to be a rotating one? Planing on scraping less than 50 profiles a day
Hi! Im kind of a newbie on web scraping so i wanted to know what do you mean by "making a living" from web scraping? Im not aware of a specific job that does only web scraping or do you sell the data you collect? and if so how do you do it, and is it legal? thanks for the video
Is it also possible to to watch a subreddit and all posts in it? And when a new post was created or a new comment appeared in a post in this specific subreddit I can scrape the data and get it in my CRM like clickup?
Also wanna ask about how yo scrape pages without pagination? Is it any other technique? Actually I wanna get tweets from X and it is with infinite scroll
Hi Adrian, thanks for the video; it was really helpful! However, I could only extract the last 26 posts from a Reddit user, even though I can see they have posts going back several years. The data I retrieved only goes as far back as December last year. Did I miss something?