Тёмный

Python & MITMProxy: Web Scraping Secret iOS App Data 

Make Data Useful
Подписаться 32 тыс.
Просмотров 28 тыс.
50% 1

Using Python we build a simple Goodreads webscraper using
- MITMProxy to observe the web traffic and get the API calls
- Requests to perform the API call in Python
- BeautifulSoup to convert the XML data
- Pandas to take the converted XML data and create a CSV file
Today's question came from a stranger on the internet over at Reddit - / trying_to_make_a_simpl...
Thank you for watching!

Опубликовано:

 

1 окт 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 67   
@thewilltejeda
@thewilltejeda 3 года назад
I was pretty psyched when this popped on my feed That mitmproxy thing is awesome ! Looking forward to seeing more of your usage of mitm and what other tricks you've picked up since your last videos
@MakeDataUseful
@MakeDataUseful 3 года назад
Thanks Will, yeah MITMProxy is REALLY cool! I can't wait to share more app API fun.
@the-web-scraping-guy
@the-web-scraping-guy 2 года назад
Great tutorial. Love that you keep your mistakes and errors there. Very helpful
@Alhamdulillahalways365day
@Alhamdulillahalways365day 3 года назад
you are the best :D
@MVGaming77
@MVGaming77 Год назад
How did you configure your iphone to listen to the proxy server? i could not
@MakeDataUseful
@MakeDataUseful Год назад
It can be a little tricky, once I had the proxy running on the computer I set the phones proxy to the computers ip with port 8080 then I navigated to mitm.it in the browser which took me to a local site hosted by mtimproxy to download the certificate. Oncce downloaded I went to settings and installed and after it was installed I went to General --> About --> Certificate Trust Settings and enabled it. That last but used to stump me.
@zecool7041
@zecool7041 3 года назад
nice video, I have one request if you can fulfill it , can you put the ipynb file somewhere where we can grab it , maybe Github or something, that way we can grab some parts of the code that we cant generate automatically, like in this case where I cant export request as a curl command because I'm using windows mitmweb, thanks for the answer !!
@alendujmic2621
@alendujmic2621 3 года назад
Did you find how to get curl command on windows mitmweb?
@haleemlahresh1581
@haleemlahresh1581 3 года назад
Hello MR I am a python learner who recently started learning about webscraping I've done some scraping programs for example I scraped comics chapters from a website which allowed me to do so any ways I wanted to ask you if you can help me start working on upwork by giving me tips on how I write my title and overview since I haven't worked before nor done any projects again thank you for you're content I find it very useful and helping its so appreciated .
@milandemooij1749
@milandemooij1749 2 года назад
Hey man! Can you maybe quickly explain how you set up your iPhone mirroring on your MacBook and how you connected it to mitmproxy?
@888felipe
@888felipe 3 года назад
Very nice this MITMproxy. Could you show how to use it to figure out how to login to spotify desktop app or mobile app, login and create a playlist using python. Thanks a lot.
@MakeDataUseful
@MakeDataUseful 3 года назад
Oooh good challenge! Let's add that one to the list... Here's hoping it's not a fail video!
@888felipe
@888felipe 3 года назад
@@MakeDataUseful why would you fail. Your skills are impressive Learning a lot with you. Thank you.
@YatriTrivedi
@YatriTrivedi 3 года назад
Great series, glad you're back! I'd love a more in depth video on mitmproxy! I'm looking to start using* home assistant and I'd love a way to interact with my smart home devices and automate them in ways they won't allow me to out of the box.
@YahiaHegazy
@YahiaHegazy 3 года назад
Informative, educational and funny!
@MakeDataUseful
@MakeDataUseful 3 года назад
Thanks Yahia!
@RidwanurRahmanextreme
@RidwanurRahmanextreme 3 месяца назад
Hi, How did you add your device with your laptop please?
@siddharthkataria5650
@siddharthkataria5650 Год назад
Please make a video on setting up mitm proxy
@eclipsed7209
@eclipsed7209 3 года назад
please do more
@MakeDataUseful
@MakeDataUseful 3 года назад
New content coming soon!
@thatoneguy1821
@thatoneguy1821 3 месяца назад
Are you familiar with the DoorDash driver app? Is it possible to scrape data from that using MITM. Specifically, what I’m hoping for is a way to scrape all of my earnings from my time as a DoorDash driver (orders, stores, dates, base pay & tip). I’m not sure if you’ll even see this, but thanks in advance. I really appreciate these videos. Excellent content!
@MakeDataUseful
@MakeDataUseful 3 месяца назад
Hey! Maybe!! There’s a couple of considerations 1) if you can see the data on screen you’re likely able to see it in an API call and 2) if they are NOT using ssl certificate pinning. Pinning is where the app doesn’t trust certificates on the device and uses its own certificates. There are ways around it but require some extra steps. Shoot me an email youtube@makedatauseful.com.au and we can explore a little more!
@Sodiumcola
@Sodiumcola 3 года назад
Welcome Back, I missed these videos!!! Keep it up. Thank you
@MakeDataUseful
@MakeDataUseful 3 года назад
Thank you! Great to be back sharing :)
@tracingisdrawing
@tracingisdrawing 3 года назад
Just wanted to say thanks for these videos! I've only been learning Python for a week now, but you do well enough job of explaining your process that I can still follow along on concepts I haven't even gotten to yet. It definitely helps with getting me primed for the things ahead. Much appreciated!
@Zancb
@Zancb 3 года назад
Great video! I found you off of Reddit today. Love your production!
@swaroop6418
@swaroop6418 3 года назад
Explained very well 👏
@Thegillmor2010
@Thegillmor2010 3 года назад
have been looking for this for ages.. great tut... don't know python but managed to follow... The app I want to scrape requires a login also please can you show how to automate looping through the data (you mentioned it at 22:35) thanks again
@s3bbe1337
@s3bbe1337 3 года назад
To loop through all the pages, find the place in the code where he sends the "requests.get()" and put that inside a loop. Also include the params object and just increment the pages value for every loop.
@s3bbe1337
@s3bbe1337 3 года назад
A tip is to add a time.sleep(5) or something at the end of the loop so it doesn't spam the API and get you banned.
@kennykenken4567
@kennykenken4567 Год назад
Does this work on windows with iPhone?
@sassydesi7913
@sassydesi7913 3 года назад
Here from Reddit. Can you tell if this can be used on teamblind.com? They have an infinite’s roll with encrypted payload. So difficult to grab data from that website.
@MakeDataUseful
@MakeDataUseful 3 года назад
Oooh encrypted payload, I do love a challenge. Let me check it out and get back to you shortly!
@MakeDataUseful
@MakeDataUseful 3 года назад
okay good news, someone has done a little digging and uncovered how they are encrypting and decrypting the payloads www.mdeditor.tw/pl/pQkk This is written in JS but it wouldn't take much effort to replicate in Python.
@oscarmartinezbeltran
@oscarmartinezbeltran 3 года назад
!Great great Content, Thank you! Suggestion: Working with Facebook API and Google Ads API.
@djmill8000
@djmill8000 Год назад
How you setup mitm
@syedhyder5630
@syedhyder5630 3 года назад
Sir. I am really excited by your videos. Thanks a lot sir. I will be very much greatful to you if you suggest any course online to learn web scraping. I have completed intermediate level python learning. I am looking forward to learn web scraping. Please help me by suggesting a course to learn sir. 🙏
@hyeonoppa
@hyeonoppa 2 года назад
Is there a way to scrape the data from mitmweb using python?
@MakeDataUseful
@MakeDataUseful 2 года назад
Hey Long! I would use MITM Dump for programmatically scraping payloads with Python. It does a really great job, there is a basic example on the MITM website. Let me know how you go and if a tutorial would help.
@rahalmehdiabdelaziz8121
@rahalmehdiabdelaziz8121 3 года назад
Adam, I'm one of your fun ... where is your LinkedIn , twitter ... etc
@gosu-team
@gosu-team Год назад
Amazing tutorial, how do you stream mobile app UI on your mac?
@alexsantiago6109
@alexsantiago6109 2 года назад
Pretty awesome video. Are you still planning on making a setup video?
@SOFRADAKAOS
@SOFRADAKAOS 3 года назад
thanks for tutorial man this was insane. Best programming teaching technique i have ever seen.
@tikendraw
@tikendraw 3 года назад
Please can you make a tutorial on how to make a legit deal in upwork? As newbie l want to have a guiding hand to start the work.❤️ please.
@islamibrahim4382
@islamibrahim4382 2 года назад
Hi bro like your content al lot, wanna ask you how to scrape real time data for the example in that video like what if i want to run it every hour without being blocked, how i can manage the headers and params
@rafnishad3523
@rafnishad3523 3 года назад
I have waited long for ur video. You should try to upload these videos earlier plzz..! and keep doing this good work thanks a lot :)
@MiguelHernandez-oe8tl
@MiguelHernandez-oe8tl 3 года назад
Great video! Can you make a video to get the comments on a Facebook page?
@thewilltejeda
@thewilltejeda 3 года назад
Haven't seen you post in a while , but wanted to thank you for the vids you made. They really helped get more practical uses of scraping skills
@irfanshaikh262
@irfanshaikh262 Год назад
2:28 offcourse Adam. Will be really helpful
@nevetsspence7740
@nevetsspence7740 3 года назад
Excellent videos!! Your content helped me solve a problem I’ve had for ages! Looking forward to more content! Great work!
@silent.-killer
@silent.-killer 3 года назад
Loving the videos man! I was wondering if you would be able to do a video talking about some of the applications of web scraping? I've watched quite a few of your videos and you have said that Upwork pays peanuts for this kind of work so I think people would be curious to hear your opinion on how real value can be created with the data obtained while web scraping. Keep up the good work, this channel is honestly a gem.
@syedhyder5630
@syedhyder5630 3 года назад
export.clip curl @focus
@MakeDataUseful
@MakeDataUseful 3 года назад
Thank you! I almost forgot it again!! The project's open source, maybe we should build a "copy as python request" 😃
@CampbellMurphy
@CampbellMurphy 3 года назад
Hey Adam, I really enjoy your videos! The MIM technique was very cool. Have you considered making a video about accessing Financial Data on Xero? They have a robust API and can provide a free demo account to query.
@chiragdeshmukh325
@chiragdeshmukh325 3 года назад
This is so cool and satisfying to watch!!!! Awesome stuff dude keep if up!!
@nikoscuatro7251
@nikoscuatro7251 3 года назад
what is that mobile simulator that you are using in the tutorial?
@Emocar2
@Emocar2 3 года назад
great vid ...also your tone is great for this video👍🏻
@yusufyuksek41
@yusufyuksek41 3 года назад
That was very helpful. Thank you!
@CodePhiles
@CodePhiles 3 года назад
great work , you always bring new ideas to the table, Thanks a lot
@calvint678
@calvint678 3 года назад
Good stuff man. Thank you. Wish I had found this a year ago 💪🏽
@rishijoshi5808
@rishijoshi5808 3 года назад
Wow, that was amazing! Thank you soo much!
@dongeunpaeng9353
@dongeunpaeng9353 3 года назад
3:18 how did he put his phone up to the screen? does anybody know how?
@MakeDataUseful
@MakeDataUseful 3 года назад
I think he was using a Mac program called Screenflow
@icinemagr4621
@icinemagr4621 2 года назад
a noob is using mitmproxy LOL
@MakeDataUseful
@MakeDataUseful 2 года назад
Hey thanks for dropping a comment, it got caught up in the RU-vid spam filter. More MITMProxy videos?
@hummer7345
@hummer7345 10 месяцев назад
can you please tell me how to use this tool (mitmproxy) thanks.
Далее
Шоколадная девочка
00:23
Просмотров 664 тыс.
КВН 2024 Встреча выпускников
2:00:41
The Biggest Mistake Beginners Make When Web Scraping
10:21
Always Check for the Hidden API when Web Scraping
11:50
Шоколадная девочка
00:23
Просмотров 664 тыс.