Python Selenium Tutorial #6 - Bypass Detection using plugins, settings & proxies

Подписаться 7 тыс.

Просмотров 70 тыс.

50% 1

🌐 NodeMaven Proxy Provider: go.nodemaven.com/scrape
💥 Special Bonus: Use "Michael" at checkout for an extra +2GB of bandwidth.
🤖 2captcha Captcha Solving Service: bit.ly/2captchapromo
This selenium tutorial is designed for beginners to learn how to use the Python selenium library to perform web scraping, testing, and creating website bots. Selenium is a Python library that provides a high-level API to control Chrome or Chromium and Firefox or Geckodriver over the DevTools Protocol. Selenium runs non-headless by default but can be configured to run headless.
Playlist: • Python Selenium Tutorial
Code: github.com/michaelkitas/Pytho...
Join our Discord: / discord
Undetectable ChromeDriver: pypi.org/project/undetected-c...
Gecko Driver: github.com/mozilla/geckodrive...
Chrome Driver: chromedriver.chromium.org/dow...
Download Visual Studio Code: code.visualstudio.com/download
Download Python: www.python.org/downloads/
Selenium Library: pypi.org/project/selenium/
Donate
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
PayPal: support@websidev.com
Bitcoin Wallet: bc1q05j8gcnq4mzvgj603cxdc8xxck4jgnu2ljsrt4
Ethereum Wallet: 0x5e7BD4f473f153d400b39D593A55D68Ce80F8a2e
Social
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Website: websidev.com
Linkedin: / michael-kitas-638aa4209
Instagram: / michael_kitas
Github: github.com/michaelkitas
Business Email: support@websidev.com
Tags:
- Python Selenium Tutorial
- Full Course Selenium
- Python Selenium
- Web Scraping Full Course
- Python Selenium Web Scraping Full Course
#selenium #python #webscraping

Опубликовано:

2 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 151

@MichaelKitas 2 года назад

You can also check how "undetectable" your selenium script is by visiting this website, from your script: bot.sannysoft.com/

@blitzed621 2 года назад

it doesn't work :/

@user-ny4ui7ow5n Год назад

man, you are creating content which doesnt exist on my country thank you.

@hiimrain600 Год назад

Nice tutorial, thank you for this video. It really helps me in scaping web

@seomaster7991 2 года назад

Thank you for your tutorial Michael 👍

@softwareengineer8923 Год назад

Thanks for a great video,it helped a lot.

@LifeOf97 2 года назад

This is great, thanks man

@keremsozen380 Год назад

the selenium goat

@epv888 11 месяцев назад

Thanks bro, thats what I needed

@larsabi_ 9 месяцев назад

YOURE THE GOAT MY FRIEND :)

@cejayrich4035 2 года назад

Thanks alot, great work

@rcgnetworks 2 месяца назад

Thanks bro 😍 It worked like a charm

@kyei22ify Год назад

very helpful, thanks

@uplifthabesha754 Год назад

It works wow, Thank you

@user-el3mj4om2m 7 месяцев назад

thanks dude it works

@thehuman5607 11 месяцев назад

Selenium expert🎉

@blackcatdevel0per Год назад

Big thanks!

@krisnadwisetyadi2964 7 месяцев назад

thank you for this video. it really help me. thanks man

@MichaelKitas 7 месяцев назад

Glad to hear that!

@ahmedellban5748 2 года назад

Thank you, you saved me

@netting. 3 месяца назад

You are a LEGEND!

@user-mm4du2cd5k Год назад

Thank u so much!!!

@KrAsHeDD 2 года назад

Thank you for this video, really helped a lot. Could you please make a video on how to use socks5 directly on selenium python. I make use of an extension to tunnel my proxy(SOCKS5) which is stressful.

@simntimtwins 4 месяца назад

Worked for me!!! 😊

@tuyenlamchan5927 2 года назад

nice video tks bro

@parpermeter Год назад

like bro, ty!

@anurajms 2 года назад

thank you

@gayashan99 Год назад

So useful 🙃

@MichaelKitas Год назад

Glad it was helpful!

@Devillunar 2 года назад

Sadly it does not pass invisible recaptcha. However, thank you for the demonstration, couldn't understand before what to change exactly.

@jw200 Год назад

Very good tuts, thank you! Howi can use multiple profiles? Or its better just reuse cookies? Do my stuff, clear cookies, load next account cookies and so on? Im working with 150-200 accounts

@MichaelKitas Год назад

Cookies for sure then

@adityasprojects7126 3 месяца назад

You are a god damm hero

@user-uw3ki8oo4e 5 месяцев назад

Can you show how to bypass the linkedinnnn verfication using Selenium , please?

@Hacking-NASSA-with-HTML Год назад

Hi! What proxies ipv6 or ipv4 would you recommend, sir 🤔? Which ones are more suitable?

@MichaelKitas Год назад

I think ipv6 is faster, since it’s newer but not sure

@faaizaziz6430 2 года назад

Thanks. I am using the same technique but need to run it on the server. How can I use this in headless mode?

@MichaelKitas 2 года назад

To run chrome-headless just add --headless via chrome_options.add_argument i.e.: from selenium import webdriver from selenium.webdriver.chrome.options import Options chrome_options = Options() #chrome_options.add_argument("--disable-extensions") #chrome_options.add_argument("--disable-gpu") #chrome_options.add_argument("--no-sandbox") # linux only chrome_options.add_argument("--headless") # chrome_options.headless = True # also works driver = webdriver.Chrome(options=chrome_options) start_url = "duckgo.com" driver.get(start_url) print(driver.page_source.encode("utf-8")) # b'

@apisn21 Год назад

can you make a playwright tutorial ? or compare selenium with playwright? Compare between headless and visible browser ? Thanks a lot, great video

@MichaelKitas Год назад

Sure, will do soon 👍

@mohamedbhasith90 Год назад

@@MichaelKitashi bro, how do i open specific profile in edge browser using selenium?

@kelechiihunda9042 Год назад

Nice video, please how can run Mobile emulation with undetected chromedriver?. It's doesn't seem to work with UC but works with normal chrome driver. And there's no useful information online on how to do it.

@MichaelKitas Год назад

Not sure either, but when I have time will do some research and make a video on it 👍

@zlwFan 11 месяцев назад

Can you show how to avoid detection using Selenium with the Firefox browse, please?

@MichaelKitas 9 месяцев назад

Good idea, yes will do a video in the near future

@nikospomas7621 2 года назад

Αφεντικο δεν κτλβνω χριστο απο αυτα αλλα εισαι πολυ μεγαλος μαγκας για την δουλεια αυτη

@MichaelKitas 2 года назад

😂😂😂

@DiegoAlbertoLeonLopez 2 месяца назад

I have a doubt, could i rotate the user to have a better No-detection or it would be the same?

@MichaelKitas 2 месяца назад

Good question! Really depends on the particular website you trying to bypass. If you are using the same ip address for example while rotating user, usually you will get banned or at least detected. So you need a custom solution for each case.

@ITEngines Год назад

would you recommend selenium or js for creating checkout bots?

@MichaelKitas Год назад

Puppeteer since it has some useful plugins to bypass detection and also captcha etc and it’s a lot easier to configure them in js

@ibrahimaouladabderahman9457 Год назад

Please how i can run multi instances at the same time with undetected chromedriver, I tried with threads but it doesn't work

@MichaelKitas Год назад

Will make a video on that soon

@03nomere28 2 года назад

undetected chromdriver n remote--debugging. which one is better?

@MichaelKitas 2 года назад

Depends on the use case

@zlukian Год назад

Lof U hahaha Greetings!!!!

@tzzar 6 месяцев назад

@MichaelKitas, my friend, tell me please. Is it updated google's code or it is problem from my side. After I put my email and click next I have a message : "This browser or app may not be secure. Learn more Try using a different browser. If you’re already using a supported browser, you can try again to sign in."

@MichaelKitas 2 месяца назад

It detects you are a bot, going to make an updated video on that soon. Hint: github.com/zfcsoftware/puppeteer-real-browser

@sagarsarkar5245 Год назад

Can you suggest how can we handle in Java Selenium ?

@MichaelKitas Год назад

Not familiar with Java, but maybe soon I am going to make a tutorial on that 👍

@haroldabella2041 2 года назад

Do you know a way on how to add authenticated proxies on undetectable chrome driver?

@MichaelKitas 2 года назад

It should be like you normally do: chrome_options.add_argument('--proxy-server=ip:port') driver = uc.Chrome(options=chrome_options)

@CodeRed-xj3ut 2 года назад

@@MichaelKitas not working with user pass proxy. By the way do you know how to save proxy with google login session?

@haroldabella2041 2 года назад

@@MichaelKitas how about saving proxy with google login cookies using pickle?

@sniper6233 9 месяцев назад

Cannot use the undetected_browser with the current selenium version. It is always giving an error that chromoptions has no headless attributes

@MichaelKitas 9 месяцев назад

Try this fix: use options = ChromeOptions() from undetected_chromedriver.v2, not selenium. Then, options.headless = False. This sets up non-headless mode, ensuring the browser window is visible. Compatibility between Selenium and undetected_chromedriver versions also matters, so double-check those if you're still facing issues!

@sniper6233 9 месяцев назад

@@MichaelKitas it worked. Thanks.

@chupapimunanyo2596 2 года назад

What is your colorscheme there? I like it

@MichaelKitas 2 года назад

Bearded Theme in VS code

@chupapimunanyo2596 2 года назад

@@MichaelKitas thanks very much :-)

@williams9708 2 года назад

Can it be used in headless mode? Undetectable-chromedriver?

@MichaelKitas 2 года назад

Yes, it should

@williamjackson1142 2 года назад

@@MichaelKitas when i try in headless it gets detected, do you know why?

@joel_mathew 8 месяцев назад

can we use ur proxy server or where can we get one?

@MichaelKitas 8 месяцев назад

I suggest to get your proxies from this provider: nodemaven.com/?a_aid=michaelkitas It's the one I use as well👍

@hasting8422 2 года назад

Hi! i am trying to log in to fiverr in the same way that you logged in to gmail but somehow fiverrs detection tool forces me to pass a captcha. Fiverr detects that i use selenium but i don't know how. Can you please help with how to get passed fiverrs bot detection tool

@MichaelKitas 2 года назад

Have you tried all methods I showed?

@hasting8422 Год назад

@@MichaelKitas Yes i have but they still block me or give a lot of captchas. I think they identify that i visit their page with selenium or maybe there is something suspicious with my user agents. I can not scrape their site and it is really annoying. Have any ideas that can help?

@LeandroCursinofoto Год назад

@@hasting8422 I have the same problem with another website.

@AlexeyZubkov Год назад

Any idea how to bypass CloudFlare with Zyte SmartProxyManager?

@MichaelKitas 9 месяцев назад

I have found how to bypass Cloudflare captcha, not sure about the one you mentioned, can you send me a url/website that uses that?

@AlexeyZubkov 9 месяцев назад

@@MichaelKitas docs.zyte.com/smart-proxy-manager.html

@Felixprz17 2 года назад

is there anyway to stop it from closing after the task is done?

@MichaelKitas 2 года назад

Any questions that you have the fastest way to find an answer is by “googling” so for example search: python selenium don’t close browser Here is what I found useful: stackoverflow.com/questions/51865300/python-selenium-keep-browser-open Also you can set a timeout, with time.sleep at the end of the code and make it sleep for as long as you want

@Felixprz17 2 года назад

@@MichaelKitas thank you, I tried it but it only works sometimes on Zalando and not anytime sadly

@KrAsHeDD 2 года назад

@@Felixprz17 I had this issue which made me mad because I didn't write for the program to close. If you imported unnecessary module and they are not used, it sometimes makes it close even after removing "driver.close" or "driver.quit". Try it if it works for you.

@yassinbousaadi1563 2 года назад

I actually didn't have the problem before using the undetected chromedriver module, but it changed after that. So maybe try with webdriver.

@Felixprz17 2 года назад

@@yassinbousaadi1563 yeah same for me. I found out that it closes as soon as it gets a error. I just set time.sleep(10) after the program is done. This way it doesn’t close instantly and I can see if it works

@Alex111motovlog 2 года назад

Can you make a video combinating this method with gspread? It would be very helpful.

@MichaelKitas 2 года назад

I haven’t used gspread but I will try to make

@Alex111motovlog 2 года назад

@@MichaelKitas I'm excited to see a video from you about this. Especially if it is with an example extracting titles of several products from different websites and adding them in the form of a list in Google Sheets. 🙌🏻👏🏻 Success for you! I look forward to learn it from you.

@arthurnicholas9003 Год назад

how can i run selenium without being headless on aws ?

@MichaelKitas 9 месяцев назад

Running Selenium without headless mode on AWS (or any cloud provider) requires a virtual display, as these servers don't have a physical screen. You can use tools like 'Xvfb' to simulate a display. However, this setup can be complex and resource-intensive. It's generally more efficient and cost-effective to run Selenium in headless mode, especially in cloud environments. If you absolutely need a UI, consider using a desktop-as-a-service solution or a dedicated server with a GUI Here is a relevant tutorial I made for that: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-yDspwAm9_sY.html

@alexdolas9911 Год назад

Thanks, but this method doesnt work with chatGPT.

@GenaTroy Год назад

Please make a video on how to fake browser fingerprints, especially canvas fingerprints.

@MichaelKitas Год назад

Nice idea

@GenaTroy Год назад

@@MichaelKitas Yeah! That would be really useful!

@ReverbedRetroMusic Год назад

it just open and close instantly. it gives me an error the handle is invalid

@MichaelKitas Год назад

Can you copy/paste the full error, as I am not sure what "handle" it's referring to

Год назад

How can I use it with robotframework?

@MichaelKitas Год назад

Not familiar with it, but I am going to check it out and maybe make a video in the future

@mejnuncnar9314 Год назад

how to save result in txt file?

@MichaelKitas Год назад

Check this out: stackoverflow.com/questions/33418777/write-a-line-into-a-txt-file-with-node-js

@mohammadaqdas7998 Год назад

I got an error of "Entry point Not found" Someone help please..

@MichaelKitas Год назад

Can you share code?

@saymindelilah8452 Год назад

hello how to deploy selenium on replit

@MichaelKitas Год назад

Not familiar with replit, but will check it out and make a video on that

@user-xu2rq8qp9e Год назад

How do you set the proxy user and the proxy password?

@MichaelKitas 9 месяцев назад

Making a video on proxies very soon

@svitoch_blockchain 9 месяцев назад

@@MichaelKitas wait for this video)

@redouanechkerint3312 2 года назад

I don't know why your Code doesn't work with me "gmail detected that selenium"

@MichaelKitas 2 года назад

Make sure you follow all the steps on this video as I showed multiple versions of the code which not all are undetected

@redouanechkerint3312 2 года назад

@@MichaelKitas OK thank you so much

@uplifthabesha754 Год назад

Can you Please do a video on how to bypass DataDome Anti-bot protection. Thank you

@MichaelKitas 9 месяцев назад

Few people requested that, will check it out and probably make a video for that

@terryterry1655 Год назад

Does it work on RU-vid?

@MichaelKitas Год назад

You can try, I had done it in the past, if you use your chrome and user data directory instead of chromium it should be easy

@washingtrend9538 Год назад

How do i know ip address and port

@MichaelKitas Год назад

IP address, are usually bought from providers, or can be found for free in some website (low quality ones)

@swahagaming5261 4 месяца назад

What about tiktok sliding captcha ?

@MichaelKitas 2 месяца назад

Good idea, will make a video on that

@ghjgjgjttytttt Год назад

bro how is this work with adsense be honest ?

@MichaelKitas 9 месяцев назад

Not sure what you are referring to.

@OnlyUseMeEquip 5 месяцев назад

are you buying comments on your videos and then replying to them?

@MichaelKitas 5 месяцев назад

No, they all real 😁

@Enzo_Barletta Месяц назад

What is that IP and port you used for proxy? "106.122.8.54:3128"

@ataimebenson 2 года назад

I used your exact code and still got detected, maybe Google has found a way to block it

@MichaelKitas 2 года назад

You can get detected by your IP address, cookies e.t.c But the ways I show help make it look like it’s more a human that selenium

@minhnhat5914 Год назад

can u share your theme name

@MichaelKitas Год назад

Bearded Theme - Arc

@rubigarcia431 5 месяцев назад

dejo de funcionar el dia de ayer

@nerffguy 3 месяца назад

el que?

@MichaelKitas 2 месяца назад

¿Dónde dejó de funcionar?

@shakils1921 Год назад

it just open and close instantly.

@MichaelKitas Год назад

Can you share code?

@shakils1921 Год назад

@@MichaelKitas thank you for your reply sir, It's now ok after updating chrome into 106 version.

@activeimmat5059 Год назад

@@shakils1921 Hey, i have the same issue, how do you uptade chrome to a lower version ?

@shakils1921 Год назад

@@activeimmat5059 that was not lower that time when I commented. But I recommend you to use chrome portable for selenium. Because it’s not update and always same chromedriver.

@phuoctainguyenlang5668 10 месяцев назад

@pduyet97 Год назад

how can I bypass cloudflare in selenium java :((((

@MichaelKitas Год назад

I haven't used java, either any cloudflare experience sorry

@abdoahmed1231 2 года назад

I have a question please When I use a proxy server and I follow people on Instagram, the follow-up is not done, for example, I have 5 followers on Instagram and I request to follow 100 people, that account is not done, they are still following 5 people, I just want to know how this happens, can the effect of the reverse proxy or forward proxy or requests sent can be redirected please help me

@MichaelKitas 2 года назад

Instagram blocks your account from following other users, like e.t.c when they see you have been spamming that action. Also, I am not sure how you are requesting for the follow, try following just 1 and see if you have that issue. You probably need to put a timeout between each follow so instagram doesn’t block you. In that case changing ip address doesn’t matter