Python Requests login and persistent sessions tutorial 🔥: the "Hacker" way | Python web scraping

Подписаться 16 тыс.

Просмотров 116 тыс.

50% 1

In this Python Requests login to website and persistent sessions tutorial I'll show you how to use the Requests Python library to login to the website, that has a protection and redirects a User after login.
So the main topic of the Python Requests login tutorial is using of Requests Python library to login to the website, using of sessions and cookies.
It's a simple Python Requests Tutorial, that includes the following topics:
- how to login to the website with the Python Requests library only (without Selenium)
- how to bypass the website's protection
- how to scrape data with the BeautifulSoup library
- how to use Python Requests session
I made this tutorial for my subscriber, that asked me to create a Python web scraping tutorial about using the Requests Python library to login to the website, that has a protection from bots.
The Requests is a Python library, that is built upon the Python urllib3 package, and it's a great tool to perform requests to websites.
By default the Python Requests login feature is enabled. We just have to provide login and password data to `auth` parameter. Also Python Requests uses session and cookies.
Also in Python Requests Login tutorial I used the BeautifulSoup library. So at some part this video is a BeautifulSoup tutorial too.
ATTENTION:
in latest Chrome versions, you cant see the form data section, instead you can find the parameters under Payload tab which is next to Headers (many thanks to Nasrullah)
Follow me @:
Telegram: t.me/red_eyed_coder_club
Twitter: / codereyed
Facebook: redeyedcoderclub
✴️✴️✴️ Web Scraping course ✴️✴️✴️
is available via Patreon here:
/ red_eyed_coder_club
or its landing:
red-eyed-coder-club.github.io...
➥ Please ➥ Like ➥ Comment ➥ Subscribe
✴️ More videos about webscraping with Python, Requests, BeautifulSoup ✴️
🔹Python BeautifulSoup tutorial for beginners
• Python BeautifulSoup T...
🔹Scraping dynamic JavaScript/Ajax websites with BeautifulSoup
• Python Web Scraping Tu...
🔹Ebay web scraping with Python and Beautiful Soup: demand research
• Python Ebay Scraping T...
✴️✴️✴️ PLAYLISTS ✴️✴️✴️
🔹Django 3 Tutorial: Blog Engine
• Python Django Tutorial...
🔹Kivy Tutorial: Coppa Project
• Python Kivy tutorial #...
🔹Telegram Bot with Python (CoinMarketCap)
• Python Telegram Bot Tu...
🔹Python Web Scraping
• Python Ebay Scraping T...
➥➥➥ SUBSCRIBE FOR MORE VIDEOS ➥➥➥
Red Eyed Coder Club is the best place to learn Python programming language:
Subscribe ⇢ / @redeyedcoderclub
Python Requests login and persistent sessions tutorial: the "Hacker" way | Python web scraping:
• Python Requests login ...
#python #pythonrequests #pythonrequeststutorial #redeyedcoderclub #requestspython

Опубликовано:

6 июн 2020

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 198

@EnglishRain 4 года назад

This was absolutely mind-blowing. Awesome job, man, SUPERB video. You're a wizard!

@RedEyedCoderClub 4 года назад

Thank you very much! Glad you liked it.

@luccarodrigues781 3 года назад

Thank you so much for this video! Helped me a lot with my web scraping adventures. Keep up the great work, man!

@RedEyedCoderClub 3 года назад

Thank you! I glad, that my content was useful for you! Good luck!

@abgezocktpfui1134 2 года назад

PERFECT explained! - especially at the minute 7:55 with the doChallengeResponse part with the mechanism to analyze and reconstruct the md5-coded hash. Good visibility of the font size and the speed of explanation.

@RedEyedCoderClub 2 года назад

Thank you for you comment, and have a nice day :)

@gisleberge4363 2 года назад

Real clever how you solved the "mysterious challenge", great content 🙂

@RedEyedCoderClub 2 года назад

Thank you very much!

@dmitryk1866 4 года назад

Thanks a lot! Very interesting and useful video. I didn't think that it would be so unusual. Funny conclusion -- "There is no security at the frontend"

@RedEyedCoderClub 4 года назад

Glad to help, and thanks for you suggestion - very useful!

@AK-zh1co 2 года назад

Excellent video and tutorial. I really like how you walkthrough your troubleshooting and problem solving approach.

@RedEyedCoderClub 2 года назад

Thank you

@shortcuts9005 2 года назад

Олег спасибо, хороший материал приятная подача 👍

@RedEyedCoderClub 2 года назад

Рад, что понравилось

@y.agamer3987 Год назад

The best video about requests on youtube

@RedEyedCoderClub Год назад

Thank you very much!

@DeepDiveinUniverse 3 года назад

The conclusion was good haha

@usingvancedplzdontban1128 4 года назад

Duddddeeee!! You make videos on awesome topics! Thanks A Ton man! Please try and make Docker + Selenium/Firefox/Chrome in the future if you can, thank you!

@RedEyedCoderClub 4 года назад

Thank you! Ok, I got your suggestion.

@usingvancedplzdontban1128 4 года назад

@@RedEyedCoderClub thank you dear friend.

@streamdr1499 4 года назад

Again, nice, thanks for sharing

@RedEyedCoderClub 4 года назад

Thanks for watching!

@sven7327 Год назад

Nice video!

@ozkankurt449 3 года назад

Red Eyed Coder Club Gerçekten çok dikkatli becerikli bir kanal takipteyim...

@jkf114 3 года назад

Great video!

@RedEyedCoderClub 3 года назад

Thank you!

@rationalpi6755 2 года назад

Great video, thanks!

@RedEyedCoderClub 2 года назад

Thank you!

@RedEyedCoderClub 2 года назад

What video should I make next? Any suggestions? *Write me in comments!* Follow me @: Telegram: t.me/red_eyed_coder_club Twitter: twitter.com/CoderEyed Facebook: fb.me/redeyedcoderclub Help the channel grow! Please Like the video, Comment, SHARE & Subscribe!

@EA-br9ii 2 года назад

You're amazing. Thank you very much for sharing

@RedEyedCoderClub 2 года назад

Thank you!

@brunonishimura 2 года назад

Hi from Brazil!!! great job

@RedEyedCoderClub 2 года назад

Thank you!

@RedEyedCoderClub 4 года назад

Let me know what topic I should cover next

@JJBoi8708 4 года назад

Can you make a video on how to buy products from a Shopify site? That would be so awesome!

@rakeshkumar-jk4lt 4 года назад

@@JJBoi8708 lol

@TipsyShort 3 года назад

Hi what if there is no id value

@stealthseeker18 3 года назад

Can you do web scraping if the website behind cloudflare version 2?

@asifiqbal12141286 3 года назад

please add the code in description.

@snopz Год назад

You helped me a lot thanks ❤️❤️

@RedEyedCoderClub Год назад

Thank you!

@yuniorbarcelochavez6800 3 года назад

great video! ...

@RedEyedCoderClub 3 года назад

Thank you very much!

@ArslanFarooqQureshi 2 года назад

This is great.

@RedEyedCoderClub 2 года назад

Thank you very much!

@thegodstudent Год назад

Thank you bro

@RedEyedCoderClub Год назад

Thanks for the comment!

@response4322 2 года назад

Can I donate to you??? Because I love this video so much! Wow you helped me solve my issue.

@RedEyedCoderClub 2 года назад

I'll appreciate for it. My Pay-Pal: molchanov.o.o@gmail.com Or you can make a donation via Patreon

@shreyaskarsrivastava9386 3 года назад

Hi, this video very informative! I am scraping a site where there is no visible function which encrypts the password, although under the Network tab, the password value looks to be encrypted. Any suggestions upon how to proceed?

@RedEyedCoderClub 2 года назад

No, sorry

@siddhartha.01 2 года назад

Thanks a lot man! After 100's of videos I something amazing! Can you scrape Userinfo(like view count followers count etc) of a Quora user without using Selenium or Requests-HTML I'm trying this since past week :(

@RedEyedCoderClub 2 года назад

Thank you! I'll try it

@siddhartha.01 2 года назад

@@RedEyedCoderClub Thank you so much, I'm waiting for your try :)

@aap204 3 года назад

Great. I have question about sending a request within a website. For example in this video we want to buy a pair of shoes. When click the buy button we can see sended order in inspect (xhr). Now, i want to learn how to send this order without clicking the buy button in python. Thank you very much

@RedEyedCoderClub 2 года назад

You need to use Selenium to perform such actions

@nygeJIbTB 4 года назад

Сразу узнал по голосу - Молчанов)

@plxgboi3602 3 года назад

)

@RedEyedCoderClub 2 года назад

Yep, it's him

@notAvn 3 года назад

i love ur english accent

@RedEyedCoderClub 3 года назад

Thank you!

@mohammadghorbani2489 3 года назад

Hi thank you so much for the video. I am wondering if I can login into a broker website and be able to place an order with python? could you please explain briefly how? or if you have already had similar videos just kindly refer me to them. Kind Regards

@albertkunz6730 3 года назад

Broker pages have higher security measures so it will be more difficult to authenticate. However many broker offer APIs for its users to automate actions. They are often called something like 'trading-api'

@RedEyedCoderClub 2 года назад

I think it's better to use API or Selenium

@NicolasMagneTang 2 года назад

Nice video, One question, for a system of login more complicates like ebay or amazon they have more complicate login protocols with tokems and encripted password. is also possible to get a way to enter with this approach? or it is better to use other method?

@RedEyedCoderClub 2 года назад

Thank you. I think it's better to use Selenium

@arete_ 2 года назад

Thanks

@arete_ 2 года назад

I would love to see more practical examples like this in the future. As you know, almost every web page is different when it comes to logins and protection. These examples are gold for getting practical experience for beginners like me. Love your work. Keep it coming and I shall hopefully tip some more. Happy new year!

@RedEyedCoderClub 2 года назад

Thank you very much! Happy new year! I'll try to create more videos like this one.

@KRock921 2 года назад

I liked how you got challenge value, but could you have made your own "challenge" value instead? If the challenge value is set on the front end and posted to the backend and unhashed there, it probably could have been any value.

@RedEyedCoderClub 2 года назад

No, fits only the challenge value that we get from frontend. I think it uses the same algorithm on backend and frontend.

@Romanticoneable 3 года назад

Thanks for your video man. I subscribed already. I'm trying to do web scraping of Amazon Kindle, but I need to log in first to access the page. I haven't found anything that could help me with this. Could you give me a hand here?

@RedEyedCoderClub 3 года назад

If nothing helps try Selenium

@Hecticam 2 года назад

If a website redirects and uses session cookies (for identification) in the main content page, is there any way of using just request library without selenium? I see the cookies already sent in the header of the very first GET request in Network tab, I can't find the way to generate the cookies , please help :(

@RedEyedCoderClub 2 года назад

In this video I used Requests.Session() class to handle cookies and keep my login stated saved between requests. Whether you need to use just Requests or you need to use Selenium depends on the particular website.

@gorabramyan 2 года назад

Hi. Thanks for video. Amazing job. I'm trying to do one with dot freight web site and its so complicated, that they don't have any form in request. Maybe you can give some ideas? Thanks

@RedEyedCoderClub 2 года назад

Try to check the requests that your browser sends to it. It can help

@multigladiator384 3 года назад

Okay, but what can i do if the post request is somehow hidden, when I sniff the traffic with my dev tools and click on the login button, the first request method is actually GET The POST request must happen somewhen before.' The Login Button has an "onLoginClick($event$) in it. I guess I need to find out what happens there to find the POST request

@RedEyedCoderClub 3 года назад

I didn't get your question. In the video I showed all action I have performed to get all needed hidden fields. For other websites steps can be different. You have to find all hidden fields on the page. Try to disable cache for requests, and check the preserve log option in settings of the Inspector.

@Nasrullah0 Год назад

in lattest chrome versions, you cant see form data section, instead you can find the parameters under payload tab which is next to headers

@RedEyedCoderClub Год назад

Thank you for the valuable comment! Will add it to the description!

@Kilogya 3 года назад

The power of automation got me back into coding. I was a VB.net programmer, but you can guess how boring that got.

@Kilogya 3 года назад

Thanks for the pin, I've never been pinned so it made my day 🙂

@thisismyescape4373 2 года назад

You are a fucking genius lol😂😂😂

@RedEyedCoderClub 2 года назад

Thank you :)

@snopz Год назад

Can you make in future videos how to scrap a website with a recaptcha without solver or selenium for free

@RedEyedCoderClub Год назад

Thanks for the suggestion, I'll think about it

@snopz Год назад

@@RedEyedCoderClub ❤️❤️

@nazarkravchenko989 3 года назад

Спасибо)

@RedEyedCoderClub 3 года назад

Thanks for watching!

@dereklee2590 3 года назад

can u do a video of getting website login details?Lately i been trying to find video how to do that all i know is do something with ip address port

@RedEyedCoderClub 3 года назад

I didn't get your question. Login details? What did you mean?

@vetrivelr5011 2 года назад

you simply superb! awesome explanation! Thanks you so much! Can we login if a website require "Soft Token" login? Basically at first time login it asks for "Login", "Password" and it re-direct us to second login page where we need to enter "SOFT TOKEN" (do change every 30 seconds). Once that successfully done it re-direct to Main page/Secure area. Like gmail has 2 way authentication login. Is that possible? Kindly help me! Must have figured this out somehow. Please please help!

@RedEyedCoderClub 2 года назад

Thank you, Redirects are supported by Requests by default. The main problem here is the getting of tokens. If you will find the valid tokens, then it's probably possible to use Requests for authorization on such websites. It's hard to say something definitely, the task requires a lot of experiments.

@gayashan99 2 года назад

i found so useful things in your channel. so i subscribed you. i stucked at copying magent link in file cr website. can you tell you me how to do that :|

@RedEyedCoderClub 2 года назад

I thinks that it can be scraped the same way

@vladislavbadovskyi1666 3 года назад

Hey, thx for awesome content, can you please make video about ajax buttons like on olx.pl (click the button to view phone)

@RedEyedCoderClub 3 года назад

Thanks for your suggestion. I'll look at it

@vladislavbadovskyi1666 3 года назад

@@RedEyedCoderClub I'm already done with it

@RedEyedCoderClub 3 года назад

Great!

@eminbedir6247 3 года назад

Hi dude, I also need help. Can you give me a hand?

@leandrosl86 3 года назад

interesting, i looking for a solution to made login in a website using request library, however, i would like if know is possible make login session using this method and call Selenium using the same session loged in the website?

@RedEyedCoderClub 3 года назад

If you have to use Selenium, why don't you want to use it to login?

@leandrosl86 3 года назад

@@RedEyedCoderClub because in specific situations, I need to press a button to activate a javascript function to load the next page. in some cases, I can get data to navigate over the direct link from the page. I long time ago I searched to resolve this problem over lib request, but I cannot found it any solution for this. Actually, exist any solution to navigate through pages? (about your question) Using Selenium method is more slowly.

@friend1439 3 года назад

Sir how to contact you? I'm trying to scrape a movies website with your code but it is returning nothing but a pycache folder.

@RedEyedCoderClub 2 года назад

You did something wrong

@tolulopeayemobola1446 2 года назад

Hi. Thank you so much for the video. I have a few questions I would like to ask. What email address can I direct my questions to? Looking forward to hearing from you

@RedEyedCoderClub 2 года назад

If you have a question it's better to ask it here - in comments.

@tolulopeayemobola1446 2 года назад

@@RedEyedCoderClub Thank you so much for your response. I have a site that I would like to try your techniques on. That's why I wanted to contact you. Anyways, as part of my graduation from a training school, I am working on an aggregator and this would require me pulling data from my own personal banking apps. This is where I was been researching and seeking help. What do you say?

@no-user-cw2md9sp9w 4 года назад

Олег Молчанов

@riam875 3 года назад

а лол это он?

@no-user-cw2md9sp9w 3 года назад

@@riam875 да

@RedEyedCoderClub 2 года назад

Thanks for comment!

@djangopython3051 Год назад

18:08 Это не weird :) просто надо было использовать `global username` но я думаю ты и сам знаешь)

@RedEyedCoderClub Год назад

а вот и нет. Откуда там global username?

@djangopython3051 Год назад

@@RedEyedCoderClub 17:59 на 22 строке ты пытаешься присвоить локальной переменной её же username = username + ':' Python ожидает, что username внутри функции будет локальной переменной. Соответственно, он ищет, где она объявляется в функции. А она не объявлена. Python падает с ошибкой. На что ты и получаешь ошибку UnboundLocalError: local variable 'username' referenced before assignment. Я понимаю что ты хотел изменить переменную username которую импортировал, но пайтон об этом не знает, поэтому ему необходимо указать с помощью global username после 21 строки.

@RedEyedCoderClub Год назад

@@djangopython3051 Все верно расписал, так и было. Но про global username мне даже в голову не пришло. Ни тогда, ни сейчас.

@moisesmasis1981 3 года назад

It would be good to know how to make the login in the background and that the system only loads the final page and not that it travels everywhere

@RedEyedCoderClub 3 года назад

I didn't get your question.

@moisesmasis1981 3 года назад

@@RedEyedCoderClub It would be interesting if you could make an automatic login that is a background and only show the web after Scrapping and that you execute it in an IIS

@_popbob 3 года назад

what if a login had a captcha? how do you bypass that?

@RedEyedCoderClub 3 года назад

You need to avoid getting of captcha. You can solve/bypass simple ones, but not Google's reCaptcha.

@jonnikwest6630 3 года назад

This approach does not seem to work anymore, and I can't figure out why... Has anybody tried it recently or have any ideas?

@jonnikwest6630 3 года назад

Forget I said anything, I was printing out the wrong variable. Subscribed! :-)

@RedEyedCoderClub 3 года назад

:-)

@tumtac4544 3 года назад

how do you web scrape a website that stops your scraping with a cloudfare protection

@RedEyedCoderClub 3 года назад

What website for example?

@capo8506 2 года назад

hello, I'm receiving this response trying to login to a website. any help please. "Invalid auth token. Please refresh the page and try again."

@RedEyedCoderClub 2 года назад

So, you have to provide an auth token. Whether the token is on the page or not - depends on the website you're trying to scrape

@TheSupernatural72 2 года назад

I tried exact same things and on my website i even have not any challenges, but the only challenge for me is im getting a cloudflare protection "access denied" cause it cant pass captcha from cloudflare. I'm using user agent for that yes. No matter how i tried, couldnt find an answer for that. Could you possibly teach that on one of your videos please? Thanks in advance.

@RedEyedCoderClub 2 года назад

It depends on the website. Sometimes the method I have shown works, sometime it is not. If the website show you the captcha (Google recaptcha) it's barely possible to solve it. All we can do is to try to avoid of getting captcha. Google's recaptcha is a quite complex thing, that determines whether you're bot or not by analyzing you browsing history and saved cookies. Probably it's better to use Selenium.

@RedEyedCoderClub 2 года назад

By the way what website you tried?

@huyhoangnguyen7465 Год назад

I want to login and dowload file with python ???

@ruiruihuang 3 года назад

what if there is no login file found in the inspect?

@RedEyedCoderClub 3 года назад

I don't get your question

@iredeyes-4351 4 года назад

Autobuy script for Realme.com as soon as the stock comes

@RedEyedCoderClub 2 года назад

Thanks for comment!

@abirhasan8315 2 года назад

pDisk.net Login & File upload tutorial please 👀

@RedEyedCoderClub 2 года назад

Thanks for you suggestion

@howiewhite4773 3 года назад

aren't you basically just logging in still but through your code? its not like your hacking in when your still uploading your user and pass in the form data...

@RedEyedCoderClub 3 года назад

Did you notice these little quotes around the Hacker word? The hacker here means the way of thought to bypass the protection. And nothing more.

@abhisheksunda7925 3 года назад

awesome!! can you make a video on how to scrap " recent submissions" from leetcoode.com?

@RedEyedCoderClub 3 года назад

Thank you! Thanks for your suggestion, I'll look at it.

@wasifhjff294 3 года назад

Amazing video. But i am fail to make fiverr.com website auto login script. Can you make a video for this. Can i contact you?

@RedEyedCoderClub 3 года назад

It's very a tricky task

@EVILMAJNU 2 года назад

Sir How To Use Google Recaptcha Same Response Token Again 👍😭

@RedEyedCoderClub 2 года назад

ReCaptcha is very tough thing. ReCaptcha uses your cookies, your search history and other stuff to determine whether you are a bot or a human. Don't know how to deal with it properly. Obviously the Requests is not the tool you need for it.

@EnglishRain 4 года назад

At 0:45 you say "without using Selenium webdriver". Any reason for not using Selenium? As per my experience, using a saved Firefox/Chrome profile with Selenium webdriver to gain access is much more simpler & quicker. So why not use it? Is there any advantage of using just requests library?

@libs1213 4 года назад

There are many advantages, most importantly Requests library is much more lightweight, which means you can run multiple scraping instances without consuming much RAM.

@andkiek 4 года назад

Selenium it's not quicker, it's so heavy and unpractical in my opinion

@RedEyedCoderClub 4 года назад

Yep, I prefer Requests wherever it's possible to use, because Selenium is slow... even headless Selenium.

@EnglishRain 4 года назад

@@RedEyedCoderClub Understood... thanks guys for all the explanations.

@VuNguyen-ni4ex 3 года назад

How selenium quicker than requests, have u tried both?

@GoodBoyDanny 3 года назад

Everytime I use selenium or requests on python on this site yeezysupply .com I get banned off the site (can't click any elements) Could you make a video on this?

@amoamo2647 3 года назад

Not that simple to bot it you will need to inject genned cookies

@mandeelev7221 3 года назад

Use user agent of your web browser coz some websites detects the bots

@iLLestTv 3 года назад

Yep. Same shit for me months ago. I was getting closer to the answer, I believe it has something to do with cookies or the user agent. Must manipulate the 2

@amoamo2647 3 года назад

They have v3 captcha as well so if your not grinning a key for that it won’t go through either

@Christian-mn8dh 3 года назад

@@iLLestTv i believe it may be you needing to add a user agent within your request. Try this

@alexdin1565 2 года назад

thanks how we can login to gmail using this method

@RedEyedCoderClub 2 года назад

I'm not sure that is possible to login to gmail using this method. I'm not sure that's possible even with Selenium. Google is too strict. This method is not a silver bullet, but it can work for certain websites.

@wasiindia8815 3 года назад

Osam! All other websites it will work But this method not working in fiver.com Please help

@RedEyedCoderClub 3 года назад

Ok, I'll try it

@wasiindia8815 3 года назад

Ok please try and make a video or contact me gmail wasisarker03@gmail.com

@wasimulti-media6421 3 года назад

Waiting for next video about this topic

@vishnudas1791 3 года назад

Could we inspect these details with any websites

@RedEyedCoderClub 3 года назад

yep

@vishnudas1791 3 года назад

@@RedEyedCoderClub which website ??? Am coding in adroid with jupyter)so it's a bit hard could please share where I can inspect these details

@vishnudas1791 3 года назад

Am asking was there any other option to inspect these details from websites

@vishnudas1791 3 года назад

I hope you got Bad English😁

@RedEyedCoderClub 3 года назад

You were asking: "Could we inspect these details with any websites". Yes, you can inspect any website. Consider to buy a cheap or used laptop - it makes you more productive, instead of using Android and Jupiter.