How to SCRAPE DYNAMIC websites with Selenium 

I made this video when I had 4 Subs. Last week rolled over 10k, thank you all!

title = video.find_elements_by_xpath('.//*[@id="video-title"]').text AttributeError: 'list' object has no attribute 'text' any ideas why that happens ? I have to add [0] before .text to throw a result.

mine is exactly the same, and no results nothing 0, and I even did import time and added some time so the page would open because it is a heavy data page and did not work, seems like this just wont work for me

may i kindly inquire why everyone is using find element by class and not by id? w/c imo is simpler and clean

Hello there; I've watched many of your videos, but none of them have any examples of product variation on the e-commerce site. Please write the link as an answer if you have a video with information about the variation? Thank you for your interest and relevance.

this is by far the best tutorial on selenium and a cool tip on panda!! Thanks @John

Awesome video! One note: find_elements_by* methods are deprecated by selenium. The new method is find_element(), with an additional argument specifying the source and type (e.g. 'xpath', 'class name, etc). Additionally, I can't seem to scrape more than 30 elements with this method, is there a reason why?

Rookie mistake: when creating the CSV file from the pandas dataframe, I accidentally put .py instead of .csv and the results ended up overwriting the entire script since it had the same name hahahaha. Luckily I was able to Command-Z it to undo.

Seriously, this is a fabulous video. You explained it very well. Please do full series of web scraping.😊

Love how you kept this video short and consise, already spent 3 hours on tutorials scraping with requests and bs4 all to discover i need to scrap with selenium for my particular site anyway

Thank you so much John. Appreciate a lot 🙏🙏🙏💜

Quick sharing of my story before I say huge, enormous THANK YOU 😊 🙏 I started a new job a few months ago and I had to do web scraping and stuff which was so new and terrifying to me. thanks to your videos I managed to go for it and do tasks which used to be completely beyond my understanding. So, John, THANK YOU MILLION TIMES for your efforts and work put into this channel

great explanation of each point, Sir please make a full series of web scraping including some advance stuff, thanks

This is good stuff... Can you get into more in depth stuff like controlling scrolls and a bit more complex js rendered page.. sites like Netflix and stuff

Agreed, this is the best, clear, practical and to the point tutorial out there. Thanks so much!

Thank you John for this, extremely helpful stuff. You explain everything so well, it makes me very excited to practice along. Also please consider recording another video showing a more complicated use case of browser automation using Selenium. Cheers !!

Can anyone please explain why import error for web driver occurs for me?

This is really helpful, thank you so much.

Hi John, I am attempting to use this to scrape prices and titles of Target products, but when I attempt to do so, I get the following error message: Traceback (most recent call last): File "", line 2, in title = games.find_elements_by_xpath('.//*[@id="mainContainer"]/div[4]/div[2]/div/div[2]/div[3]/div[2]/ul/li[1]/div/div[2]/div/div/div/div[1]/div[1]').text AttributeError: 'list' object has no attribute 'find_elements_by_xpath' could you please point me in the right direction?

Super helpful! Allowed me to finish my project for work! Thank you!

OMG that little .text is exactly what I was looking for. Thank you sir you solved my problem

this type of content is very rare sir...thank you sir

How long would this work for typically? Do websites change their divs often enough that would break this script?

Thank you sooo much for this tutorial! Can you also do LinkedIn profile scraping..?

Hello John! Thank you for all these videos. I've been working on some webscaping projects and literally been on your channel all day! I will recommend this to all my friends to subscribe! Quick question, for a website like glassdoor...what would you use for the XPATH for the company title and position? I can't seem to figure it out.

I am a total beginner in coding and I find your videos very helpful, thank you

When i set the variables in for loop (ex title=video.find_element_by_xpath('.//*[@id="video-title"]').text) it only returns for the first video. IF I run the for loop and just say print(video.find_element_by_xpath('.//*[@id="video-title"]').text) I get all the videos. Whats messing up when I set them to the variable title? I copied yours word for word

Great, straight to the point tutorial. When I include the "." in front of the xpath I get the following error: Message: no such element: Unable to locate element. When I remove the dot I only get the information from the first video. Do you know why this might be? Thanks

thank you so much! this is really helpful!

You gave us a very clear idea of how can we us selenium. Thank you brother

Could you perhaps do a video of how to use scrapy AND selenium 🤔 to scrape dynamic websites. I had a hiccup in a technical interview where I was asked to scrape a dynamic website and he wanted me to use scrapy with selenium. I've used both of them separately but never together. And I never got a call back. But it still has me wondering now how that would work.

Great tutorial. Can you also make a video about following the links and extracting data from inside the link?

why don't you use jupyter-notebook?

This video made me love Python even more ....subscribed !

Great content to learn web scraping. Not discovered by many yet, will be a huge hit thank you

Great content as always. Short. Precise and clear..

how do you determine when to use find elements by xpath vs find elements by class name?

this content is legen------------------------------wait for it--------------------------------------------------------------------------------------- darry!

playlist help us to reach easily

hi, i am getting this error AttributeError: 'list' object has no attribute 'find_element_by_xpath' how do i solve this?

If anybody has problems with your video titles/views printing, try using time.sleep(1) after driver.get(url). I assume this allows the webpage to fully load before it tries scraping for your elements. This fixed my issue

Damn! From 4 to 2.44k subscribers. Really good work !

Love this! ❤ please make more vedios on latest python modules

Went through this today. Well explained again. Using VSCode. When browser (FFox) opens there is a google account dialogbox. Caused script to timeout before RU-vid page loaded. To overcome this did import time and time.sleep(5) after driver.get(url).

Thanks for your high-quality and well_explained video! I found this by algorithm (Thanks algorithm) and your videos is so helpful. I have one question! Currently I'm practicing web crawling through youtube and I'm curious how I can crawl video length data. The ads interrupt :( plz let me know if there any good way for this! Again, thanks a lot for your videos:)

hello, i just found your channel and subscribed. On channels that have modal pop-ups for GPDR consent, etc is there any way to use requests or do I have to use selenium? When I use requests, the function never returns.

When you right clicked and copied the path it was like watching the person who discovered fire. #gamechanger 👏👏👏

from selenium import webdriver driver = webdriver.Chrome() driver.get('ru-vid.com/group/PLRzwgpycm-FgQ9lP_JTfrCa9O573XiJph') videos = driver.find_elements_by_class_name('style-scope ytd-playlist-video-list-renderer') print(videos) can you please tell why the above code is giving only the first element when multiple elements are there

good tutorial. Please show how to add the video Link too?

how would you go about grabbing the number of comments under each video? would you need to click on one video and somehow grab the html tag for comment and pass it through a loop? I've tried doing something similar with beautiful soup and im running into a wall each time I attempt it

This is great! thank you :)

Thank you for sharing this video! I'm able to start my own scraping project from a website that has grid results like in your example!

beautiful, You made me all revised in counted minutes ot time . Thanks John. Wish you can make a video to crawl through a lsrge list of URLs using supplied by an exl sheet.

So easy to understand, BEST selenium video !

Hello, Thanks for the nice tutorials. In a video you showed about the "requests_html" library to scrape dynamic content, so how feasible is it in comparison of Selenium.( I understand the Power of Selenium), but I have few questions: 1) can we scrape big sites like RU-vid, amazon with reauests_html, and get the dynamic data ? 2) how does requests_html render js in the background ? You passed sleep=1 in the html session. So is requests_html also using selenium in the background to render the dynamic js data ? Please answer my queries if you know, it can help me figure out things. Thanks very much.

Really clear, however "find" in my code gets only visible elements, but if I scrool i can retrieve more elements.... how can I scroll till the limit to get as much elements as possible? thanks

I've found that selenium can get terribly slow while locating elements especially when you try to locate by xpath. Finding by class_name or tag name are seemingly faster. Still though it took about 15-20 mins to process 2000 entries and write to file.

That's exaaaaaactly what I was looking for. Thank you mate!

Nice, do more videos, like scraping website like dynamic, noiiz sound samples downloading, please try to put a video on how to download sound samples from noiiz sound instrument

thx for this video it was very useful < keep going dude

I want to crawl all links of chapters on bigtruyenz.com/truyen/dao-hai-tac/, I found class 'wp-manga-chapter ' as you did, but it did not work in this site, could u help me, thank you so much

way to go from double digit views to tens of thousands!

Thank you so much, sir. That pandas dataframe technique is really helpful. I will share this video with my friends.

Thank you for the video, it was really useful. How would you also extract the URL of the videos?

Hi john, I have been trying this method on Facebook, but I can't seem to get it to work

great job , thank you

Good Work. Perfect level for a beginner. Easy to follow and understand. Learned a lot, thanks.

Thank you so much John.

Good Sir! What if I also wanted to extract the "href" from the video title, how can I do that?.

did the API change? I found out I have to use driver.find_elements(By.CLASS_NAME, ("class_name"))

Sir, your videos are amazing.Really helped me clearing many many doubts in scraping.Thank you so much.May God bless you.!

How can I use Python to search for a list of serial numbers in a document column, employ a search toggle (similar to the RU-vid search toggle), and subsequently extract the results obtained for each serial number?

I'm trying to scrape a webpage that loads a table in steps of 10 entries as you scroll down the page. This method doesn't load the entire html script for me. There's 2000 entries in the table. How do I force it to load all entries?

Thanks for the video sir . Is it legal to scrape yt channels ?

Tengo un robot que está funcionando. ¿Te lo puedo mandar para que lo corras, identifiques las áreas de oportunidad y cotizame cuánto costaría mejorarlo? I already have a "fully functional Chrome-Selenium-Robot" wich I want you to run in order to identify opportunities to improve it. Can I send you the script?

Homeboy! with this simple video you helped me sort out almost every question I had after hours of useless content! 10 out of 10! You just gained a new sub

Please put a video on download sound samples from noiiz website. By instrument

thanks for tutorial awesome

Hey John, I hardly ever comment on videos, but I wanted to let you know that this was exactly what I was looking for! And only 11 minutes? Great job, and thank you!

what version of selenium were you using?

Amazing tutorial

How do you click on link elements such as "

A big thankyou sir for helping out with this! I was able to follow through your video step by step, however, the output gave only 30 entries and the page that I used as a URL has more than 100 videos. What could be the reason of it, sir?

Thank you subbed.

Hi, I have one problem, actually why I'm getting no such element exception. I inspected the elements. I tried XPath, CSS selector, id to get that element but still getting that exception. Can you help me with why that exception occurs?

Sir can I know which type of data that client usually ask for.

Thanks man you helped me alot

man, how happy when I found your channel, very clear explains, this is the true gold mine for me!!! Thank you!

bro you are awesome!

Great video, I was following along for my url and get a successful run in my for loop but no output is shown even though I have it printing my output. Have any suggestions for this?

Great job! Could you have a video to guide through scrape booking reviews :>

Anyone who's been able to execute this recently , can you please help me out, many dependencies have changed and functions no longer operational, foe example how to do i get the text info and not all the object info, cause .text no longer works

I'm getting same data all the 30 times... please help me out where I'm mistaken

Wonderful! Thanks man! I especially liked how you started from each video 'catalog' and then iterated inside for each video. In the future, would love to see more complicated examples for example, how to go to a window, collect data, close the window and things like this...

Hello, I am having issue when i run the code everything seems to be working fine but dataframe is empty. Any reason why this would happen?

Nice video but how can I get the results into excel with the link of the thing you are trying to scrape?

Thank you for this very useful video!

Thanks for sharing 🙏 But I have a question about selenium, namely can selenium create an account on a site like Instagram with a total of 10+ ?

Very good video. A question. How to block the browser exe screen?

how can we monitor and get data from dynamic graph available on website.For eg=> in.tradingview.com/chart/