Who is FASTER? Scrapy vs GO vs ME 

So... Basically for python if execution time is important - Async httpx, if speed of development - Scrapy. Cool to see such tests. I didn't know that Scrapy is so much slower.

Scrapers built with scrapy and custom asyncio based solution presented here - are not (logically/algorhythmically) equal and from my point of view It is not fair to compare it directly as here. On 4:01 we see that custom asyncio based solution has get_total_pages method: After processing of first catalog page - scraper will send request request to all remaining catalog pages. While on parse method in Your scrapy solution 0:43 we see only one (next) catalog page request created per processed catalog response. it means that request to page4 will be created/send only after processing of catalog page3; page5 - after page4 and etc. It will cause additional delay and it will increase idle time of scrapy application and this part is logically different comparing to Your custom asyncio based code. However if implement this approach in scrapy and adjust it's concurrency settings to: "CONCURRENT_REQUESTS": 150, "CONCURRENT_REQUESTS_PER_DOMAIN": 150, And (the most important) "LOG_LEVEL": "INFO" Result performance of this (at least in my tests it dropped from 25 secs to 11-13, I mean "elapsed_time" stats parameter in scrapy) may look closer to other presented options. It is not something unusual or unique in scrapy. I quite frequently see application of this approach.. in stackoverflow questions (scrapy tag) and on other scrapy community channels - implemented in start requests (with hardcoded number of pages inside cycle) or by setting in start_urls as list comprehension (f string with indices from range..)

Hi John, have you considered making a video on scraping and downloading images using scrapy (if you haven't already). I'm having a hard time in this area

Oh, htop on the background, fellow Linux user.

Title should read "who's"; not "whose".

Hi how can I use scrapy to scrape a html response I get from using scraper api

You wouldn't happen to have a discord channel or some form of community place would you?

How about RUST.

Hello John, i hope your fine, just i want to know how can i resolve (reCAPTCHA - Google)