Check out ScrapingBee for youself here: www.scrapingbe... Scraper API www.scrapingbe... Patreon: / johnwatsonrooney Donations: www.paypal.com... Hosting: Digital Ocean: m.do.co/c/c7c9... Gear I use: www.amazon.co....
All good (aside from AMZN now stopping anything but 1 query - need delay?) up to prices = orm.select(p for p in Price if p.asin == item) ndexError: tuple index out of range
Hello John! I'm a big fan of your videos, and I have a question. If I create a web crawler to crawl, let's say, 1000 e-commerce websites, how can I bring that data to my website? I'm building a product search engine that gathers products from different specific clothing websites into one place. It's not scalable to scrape the products using selectors or classes because I would have to do it store by store. What solution do you think could solve this problem? Thanks!
Hey! Thanks. With so many sites you have a few options, I’ve created rule sets before to cover as many cases as possible for extracting product data. I also have saved the scraped html rather than parsing right there and then so i can easily run different parsing options against it. The other option which I haven’t tried yet is trying some kind of AI parsing option to extract the data you need.
I'm using IPRoyal and I see they allow unlimited concurrent requests. How do we tap into this whilst scraping to a database? Currently I'm adding to db one at a time. Last time I tried concurrent db writes I created a 44gig db that nearly crashed my computer. s/o from 🇿🇦
@JohnWatsonRooney so currently I'm scraping per category as a chunk. I add scraped items to dataframe and once I'm done with category I add to db. If I'm hearing you correctly, I should use concurrent to scrape the category products and once the category is done I "pause" all the scraping and add to db then "start" the concurrent scraping for the next cat?
Depending on the data you wish to scrape, here are some rule-of-thumb rules that might help: 1) have request timeouts 2) use proxies 3) change user-agents There are, of course, many more subtle things that you can use that might help, but obviously, nobody will publicly reveal something it took years to figure out.
