How to Scrape JavaScript Websites with Scrapy and Playwright 

That’s great I’m glad i could help!

If it isn't too much trouble, would you mind eventually making a video on hidden json api endpoints that require some kind of cookie or header authentication? Thank you for all the invaluable content :)

Nice video again!!

Thank you for watching !

Thanks, appreciate it. Cool user name ha

My pleasure! Glad you enjoyed it

@@JohnWatsonRooney kindly push code on github.thanks

Thanks!

Could you please show us a tutorial how to submit form and login. And how to click page and pagination. All the detail about page coroutine.

That’s great! I’d recommend playwright first

@@JohnWatsonRooney I do have one question I'm using a crawl spider do I do def start request afterr my allowed_domains and start_urls or before?

Thank you glad you like it!

Yes that would work. This just fits in the normal request/response flow which is why it’s so useful

Thank you!

Have you figured out a solution to this problem? I am having the same issue.

Do you find the solution I am also encounter with same problem on windows

Thanks! I would use playwright over splash right now for general projects - it works well and is easy to use and install (no docker needed)

@@JohnWatsonRooney yeah, definitely. But I noticed that framework donot work on Windows, so sad !

Hey, the end goal is the same but they go about it in slightly different ways. Splash is Specifically designed for rendering pages but requires a bit more setup, while the scrapy playwright integration is newer and is self contained rather than a separate service. There are use cases for both but right now I’d lean to playwright, certainly for personal projects

Yes you can, I haven’t done it in a while I think there is more setup required though

Are you using windows? Because I am also encountering this error. And after searching it was found that it works good in Linux. Windows is not compatible.

same

Hey did you try pip installing scrapy-playwright?

You have not reported whether John's reply helped. I encountered the same issue when working with Pycharm in a project venv. Resolved by having PyCharm install the package (preferences/project interpreter).

Sure I will look into it

@@JohnWatsonRooney awesome 👍

It’s not something I’ve done before, there might be issues running the actual browser - is that what’s getting stuck?

@@JohnWatsonRooney Yup exactly. Runs fine with 'scrapy crawl' command, but 'curl ....' gets stuck at the playwright part. Maybe rely on cron for now?

@@StephenForder I have limited experience with scrapyd i';m afraid, maybe yeah just run it on cron

@@JohnWatsonRooney thanks John :)

I'm not sure what went wrong on my first server setup attempt, but on my second attempt, scrapy-playwright and scrapyd are playing fine together on Ubuntu 22.04 👍

I've heard from another viewer that playwright has problems on windows but i don't have a solution i'm afraid, maybe checking out their github to see issues might help

@@JohnWatsonRooney thank you so much I guess I'll have to find alternative for JavaScript websites other than Scrapy

Yes it can absolutely do that. It would be worth check what happens network wise when you click it though, often you don’t need to do that at all and can make the same request without playwright. Check out my video on hidden APIs (best scraping method)

@@JohnWatsonRooney thanks for the suggestion, I saw the video on the API but there doesn't seem to be anything I can use.

I didn’t actually try that! I would expect yes and we can pass it into the meta arguments at the top

Yes you can using the setting PLAYWRIGHT_LAUNCH_OPTIONS = {“headless”: False}

I think this is gruvbox material, or gruvbox dark medium

I only would use this as a last resort, I would explore other methods first, html scraping, reverse engineering the API. This method is good where time isn't a requirement

Oh doesn’t it? Sorry I didn’t realise I’ve used either Linux or WSL in windows for years for development

me too

Did you manage to fix this? I'm having same problem

scrapy-playwright is not working on windows. you can try it on linux.

@@oktayozkan2256 me too, windows dont supported

@@valostudent6074 if the problem persists, wsl (windows subsystem for linux) could be a good solution.

hey its Python 3.9.7 with a virtual environment using venv. I've done a similar thing to this recently on 3.10 without issues as well

@@JohnWatsonRooney thank you very much for the answer, I was working with version 3.8 and I did not know whether to downgrade to 3.7 or move to 3.10 at once, have a nice day or night.

@@RicRod no worries, try 3.11 if you can it has some improvements in it

​@@JohnWatsonRooney This does not work on windows with vscode + anaconda. Did you use Mac pc or Linux pc for this? Thanks

The error like that `Traceback (most recent call last): File "C:\Users\Future\AppData\Local\Programs\Python\Python39\lib unpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Future\AppData\Local\Programs\Python\Python39\lib unpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\Future\Desktop\venv\Scripts\scrapy.exe\__main__.py", line 7, in File "C:\Users\Future\Desktop\venv\lib\site-packages\scrapy\cmdline.py", line 144, in execute cmd.crawler_process = CrawlerProcess(settings) File "C:\Users\Future\Desktop\venv\lib\site-packages\scrapy\crawler.py", line 280, in __init__ super().__init__(settings) File "C:\Users\Future\Desktop\venv\lib\site-packages\scrapy\crawler.py", line 156, in __init__ self._handle_twisted_reactor() File "C:\Users\Future\Desktop\venv\lib\site-packages\scrapy\crawler.py", line 343, in _handle_twisted_reactor install_reactor(self.settings["TWISTED_REACTOR"], self.settings["ASYNCIO_EVENT_LOOP"]) File "C:\Users\Future\Desktop\venv\lib\site-packages\scrapy\utils eactor.py", line 66, in install_reactor asyncioreactor.install(eventloop=event_loop) File "C:\Users\Future\Desktop\venv\lib\site-packages\twisted\internet\asyncioreactor.py", line 308, in install reactor = AsyncioSelectorReactor(eventloop) File "C:\Users\Future\Desktop\venv\lib\site-packages\twisted\internet\asyncioreactor.py", line 63, in __init__ raise TypeError( TypeError: ProactorEventLoop is not supported, got: `

@@KhalilYasser me too

Yes it is, it does depend a lot on the sites protection and how many requests you are making

AttributeError: 'PipeTransport' object has no attribute '_output'

Did you manage to fix this? I'm having same problem too

@@adamdavies1956 Scrapy-Playwright now only works on Linux/ Ubuntu. I setup ubuntu then I managed to run this

@@raisulislam4161 Hey Raisul, how do you know it only works on Windows?

Selenium is still used very widely but it’s great to have an alternative and playwright has been brilliant to work with

Yes but it will be installed by playwright

Absolutely. I do my best to show you the method but what you don’t see is the hours and hours of learning I did when I was new to it. It’s not easy but every time you overcome and error you learn what it was and why it happened and learn the insides of the errors. I promise if you keep working at it you’ll get there.

Oh really? I thought it would but I don’t use windows anymore to check

Work smarter not harder!