Very helpful. Many other tutorials were not helpful with integrating proxies into scrapy, but this was exceptionally clear. ScraperAPI is very easy to use too.
Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?
I've gone through several of your videos and picked up some great tips, thanks! Since you seem to be heavily focused on scraping, I wonder if you've looked into ML-based scraping. Instead of manually coding heuristics, let the machine learn how to find the relevant data!
The rotation feature provided by Zeus Proxy allows me to change IP addresses seamlessly, maintaining a dynamic online presence that reduces the likelihood of being flagged by platforms like Twitter.
when i implemented the same code and ran it. It is displaying different websites for proxies (IP obtaining purpose) but it is printing same IP every time. Could you please guide me what is the reason.
I'm always getting this error 2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] WARNING: No proxies available. 2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] INFO: Try to download with host Please help
Is it fair to conclude that some some web servers are just so adamant about not getting scraped, and good at implementing it, that no amount of coding is ever gonna get past their security?
@@codeRECODE I get this, it only changes on the 5th attempt: { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101, 207.74.82.103" } I've checked and triple checked my code matches yours. Any suggestions?
and that's precisely the reason I recommend getting one of the cheaper proxy providers. I will see if I can find something else, but sooner or later, they will all stop working.
@Ataime Benson - solving Recaptcha is a sensitive topic. The presence of any sort of captcha means that the site is discouraging from being scraped. For _educational purposes_ I can tell you that there are many services like 2captcha that can solve Recaptcha for a very small amount of money.
Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?
![Code [RE] Code](http://yt3.ggpht.com/egxpMo3Rdbr2eIHscL0XaiaA5WXvlZtQ9AADpLW-dPSCaO8QBzvjog4dQKpO61T_B3q2iZcCyA=s48-c-k-c0x00ffffff-no-rj)