In part 2 we auto-login with Selenium then use Python BeautifulSoup to scrape the contents of the pages to create a Pandas dataframe at the end. Part 1 available here - • Mastering Browser Auto...
Thanks for these tutorials, you have an excellent teaching style. I look forward to sharing your videos with my students as supplemental materials (and "scraping" them for potential assignment ideas). Keep up the great work!
Great video! I love your example use of Python code in each of those sections. Very helpful in visualizing the code being executed and the data being returned. Thank you very much for putting these together!
Just stumbled across your channel and this is some of the best content I have seen related to Python! Your teaching style is fantastic. I hope you keep making these videos. It would be cool if you explored making a video on using Selenium, scraping data, adding that data to a postgres db, making a UI that displays that db data. Oooor another thing I've been wanting to try is taking the data I scraped and adding it to an Airtable sheet for a quick and easy/shareable visualizer. Anyways, keep it up, really enjoying it!
Thanks Will! I'm yet to get using Scrapy but I'm thinking about putting a short series of "learn with me" videos sharing how I approach learning new packages and techniques to help make new topics stick.
Hi! Your videos are great, it helps me a lot. But one thing I still can't completely understand is how to work with network page in DevTools, how to pick an element you need etc. Would be really interesting to get a little bit deeper. Watched all videos so far and still couldnt find explanation.
Thank You for bringing high-quality education with super easy conceptual techniques for all levels of learners. Just a thing, where someone should ask questions for a specified problem regarding web scraping? Thanks
The part where you described calling the function login() and process_products()...Does this work in python IDE because when I use similar code in pycharm by getting all the code from jupyter notebook, it doesn't bring anything when I call login function
Just came across your channel and went through all the videos this week. Top notch content mate. I noticed you haven't uploaded anything new for a while. Hope all is well. Will you resume uploading new content anytime soon?
Omg! Such an awesome video! Quick question. Selenium has a known issue with drag and drop function. Do you know a work around for it? I've been stuck on this isssue for two months now.
Hi, very useful! If I have java script content to scrape, how do I keep going instead of BS4, can you past something? I mean, in the html = driver.page_source and after bs4.. how do I do that to keep scraping with selenium or another language as helium for JS?
Hi sandeepyadav1478, of course I can but first you'll need to provide a little more detail. You mentioned logging into Gmail, is there is a specific task you are looking to do once you have logged in? Also, what code/packages are you working with? Knowing these two things will help me tailor my response to your specific use case.
@@MakeDataUseful Ahh, i want to find all mails send to by 1 particular person. and i m using python with webmanager, bs4 and selenium same as u r using expect IDE, i m using sublimetext 3. but i can't login in my gmail account coz it didn't took my keys(userdata) .
Hey, i have encountered 2 more problem [not from the video code]. 1. How to take values of text changing div. (like after click it changes value) 2. my jupyter nootbook works different in every cell, like if i wrote imports 1st cell then i have to write in 2nd cell too. so i can run code in different cells like u. Thanxx
Great video. Spent half a day reviewing other videos of similar content and you went through it with the Kiwi accent plus straight forward approach. My challenge is after I login, my website redirects to the 'data' page. When I try to "driver.get('https:\\my.html.page') then it tells me my login is not correct. There is a remember me check box, which I have ticked in the code before trying driver.get. I suspect it is cookies but not sure on next steps. Any guidance would be appreciated mate.
Hey I started using python for web scraping since I saw your video. But I have reach a mental brain freeze cuz there is a epub book web that I like. And I have been trying to scrape from it with out any success. The web is VK website is you can point me in the right way I would appreciate it
When I try to run the process_product function, I place in the html like you have done, but it spits back an empty list [ ]....I have been reviewing your code and I don't seem to see where I went wrong.
@@sandeepyadav1478 have to send a request to select the list representation, I wasted hours thinking what went wrong. Will never make that mistake again 😂
Adam, big fan of yours here!. I've hit a brick wall. Trying to scrap pages over an API call; first page is no problem. I can't find a pages param in the API URL call. Any work around you can suggest?
Thanks for the feedback! Is the API one that appears in the network tab in chrome? If so, can you navigate to the second page in the browser and see what the API call looks like?
@@MakeDataUseful Thanks Adam. I'm getting syntax error when I insert page variable like so {page} inside the search query url . I think it's my bad...let me sleep and recharge. ps! let me know if they is a way to share my code.
