This script I threw together saves me hours. 

Fantastic “apprentice” content. This assumes a basic understand but also pushes the novice forward. I really appreciate it!

I didn't even know that selenium can be used like this, thank you very much, great work as always))

I used seleniumwire for create a scraping bot. It’s a very good package to grab the backend requests. What i did was using selenium i logged-in than grab the cookies and the backend api ;) than i simply closed the browser and used the python requests lib to make the request to make thing little bit faster. Eventually, i dockerized everything and than i have this container image which i than pushed on aws ecr and run parallel on aws ecs. Pretty amazing.

This is really great, and a great foundation, too. I can see this being extended to support so many things, too.

"Because. I. Don't. Care..." 😂😂

This is really good thank you so much for this tutorial

Hello thank you for the amazing video. Wanna ask please how can I bypass 403 forbidden, for cloudflare when I am requesting an Api? Thank you for all your efforts 🙏🏽

Love your thought process behind writing this! It makes it easy to follow why you do a certain step, and if it’s necessary for others! Great vids keep it up!

great video. thank you john❤

How did this work without importing 'requests'?

golden content here

Hi John!! I really appreciate this new content. I have a query to ask. I was using selenium webdriver in chrome to fetch data from a website. The script is working just fine but after certain iterations, the driver is not working properly or the way it should. I am getting a NoneType error. I tried clearing the cookie and starting a new session and then continue from where I left off but it is still not working. Any suggestions on this?? I really appreciate it!! Thanks!!

Amazing content thank you

Can it make complete copy of requests with url, headers and payload?

What if api only called when any user action occurs then?

thank you, i am wondering if you wine money with this tools ????

Are you no longer using neovim?

I don't really get it.. I mean you can filter Network tab by link or a word "api" too if you want to. Plus this solution will not work for everything, but Network tab will. Other than filtering only needed requests this solution doesn't seem to do anything. And yeah, you can do a bit more advanced filtering here, but.. Does this really saving a lot of time for some kind of task? It's just hard to see how for me. Did I miss something? I'm making AJAX scripts dealing with forms for the past year+ and for me it would be absolutely useless.

is this better than pupeteet network events?

Hi, thanks a lot, this was very helpfull to learn. I use contextlib.surpress, its actually faster than try/except and it looks better i think. Your function would look like this: import contextlib for request in driver.requests: with contextlib.suppress(Exception): data = decodesw( request.response.body, request.response.headers.get("Content-Encoding", "identity") ) resp = json.loads(data.decode("UTF-16")) resps.append(resp) return resps

Can I bypass hqq.tv devtool blocking using this?

Nice idea. But I will still prefer to log the requests via Network tab or Burp suite. The chromedriver detection will also kick in for some sites.

Anyone else update chrome on their pc and had all their scrapers break?😅

Hi John, big fan. Thanks for toturials ❤ I need to contact you on any social media, i need one site scrape help kindly

Hello John, could you make a video how to scrape data which a server send trough a websocket connection in live mode?

.

good video . Do you capture keys for api in Selium wire as well. As some api use session keys

complete noob here just started web scraping for some reason the seleniumwire import is giving me this error import blinker._saferef ModuleNotFoundError: No module named 'blinker._saferef' I've been searching online for help for hours. changed python versions (currently using the same one you're using in the video) nothing seems to work. please help thank you in advance

Guys, I'm watching with passion but for what it would be helpful? What are web-scrapers actually doing?

that's very useful, thank you

i was working with selenium / selenium-wire until i could not debug the issue while selenium-wire is not listening the right port where selenium is running while dockerised.

So this is kinda like playwright network events right?

are you using JetBrains Mono font? If yes, then how it looks so thin?

Great work bro!! And I have one question also if I want scrape Walmart everytime robot or human pop-up comes so can you please guide me how to Bypass this type of bot detection system? Thanks and love your content because of you i learned python!! 👍

Are you using arch linux sir ? And thanks for the content ! 🥰

can you please share the script that you created for my future reference ??

What is the vscode theme and the font used in this video?

Amazing, just something I was looking for. Need to look into more if I could fetch all the IPs too

Hi John! Love your work. Could you share the codes of your videos.