Always Check for the Hidden API when Web Scraping 

I've doing this for years as a self taught programmer, there are some little tricks you did here that i didn't know, thank you for the video.

I've been using this trick for a while now, and I've learned it from you, so thanks. Amazing work man

Nice video. It’s worth noting as well that many APIs will pageinate, so rather than checking how many total results exist and manually iterating over them - you just check to see if the ‘next page url’ or equivalent key exists in the results and if so, get that too until it doesn’t exist anymore, merging/appending each time until the dataset is complete 👍

I rarely praise anything, but this tutorial was SO good! Well explained, no filler. In 7 or 8 minutes you guided me through finding the hidden information I needed, which tools I need to use and how to automate it. This tutorial gave me enough confidence to try to write my first Python script! Within hours I built a scraper that can pull all metadata for a full NFT collection from a marketplace. Without this video it would have taken days/weeks to discover all of this

Please ignore my first comment. I checked out your first video in this series and learned about using scrapy shell to test each line of code. With that I found the bug in my code. The code worked PERFECTLY as advertised. Your the man! Much thanks!

Loved everything about this video! Great delivery style, production quality and interesting topic for me. First time visitor to this channel and not a Python user (thanks, RU-vid, for your weird but helpful predictive algorithms).

Wow, thanks for this excellent tutorial! I just spent all this time writing cumbersome Selenium code, when it turns out all the data I was looking for was already right there!

bro, you're a game changer and i love you. if i ever see you in person ill offer to buy you a beer, or lunch, coffee whatever

This video was the answer to my prayers! The next best option was to watch an one hour video and hope they would teach what you taught... In 10 minutes!!! 👏👏👏

This video came into my feed just a couple days after I used exactly this method to collect some data from a website. Very good info! This is much easier than web scraping. Unfortunately, in my case, the data I could get out of the API was incomplete, and each item in the response contained a URL to a page with the rest of the info I needed, so I had to write some code to fetch each of those pages and scrape the info I needed. But much easier than having to scrape the initial list, as well.

This is a true gold nugget. Thanks for demonstrating how to easily view the request in Insomnia and auto-generate code!

Awesome advice, a lot of people skip checking the requests when building scrapers but it can save a lot of time when it works

I just want you to never stop creating such informative video. For god sake.

This is, no joke, the most useful video I ever saw on RU-vid!

I like how you regularly start sentences with 'you might think' assuming we are all idiots. I approve, glad smart people, like you, make time to explain to us plebs how the world works. Apprecated.

Been using python for a couple years now as a picked up language and I really appreciate getting to see how someone experienced approaches these problems

really nice and helpful tips in an actual topic with a sight-pleasuring recording quality, thank you for your time and efforts.

HUGE! I've been looking for this info for 2 days. 12 mins of your video better than anything else, by far. Thumbs up and thank you so much

I was struggling with selinium to extract a table from javascript website. This video saved so much time. Thank you

You have shown me the light. Thank you for stopping me from making more web scripts that load up web pages in browsers to click buttons.

Nice video! Used a similar method to collect European Court of Human Rights case documents since there is no official API. Glad to see such methods gaining popularity online, it’s so useful!

I have used the inspect with network method but wasn't aware of the copy as url method, thanks for that tip will save me a lot of time!

There is always something new to learn. I’ve been spending hours to grind such an information by hand-writing the whole program to get my result ;D Thanks!

Thanks for sharing! This has helped me a lot. After struggling for weeks with selenium, I was able to apply this technique fairly quickly, and am now using it as source to scrape ETF-composition data to feed directly into a PowerBI dataset. Much appreciated!

I always come to your channel for these excellent time-saving tips and tricks! Thank you!

Nice tutorial on scrapping, some tricks I have been using myself, and some others never heard of until now thx for sharing!!! Small adjustments if I may (please don't take this as criticism) I think you don't need to loop over each product to copy it to your res, you can use extend instead, also I think the header didn't change so you can take it out the loop over pages

Great tutorial. My screen scrapping job went from 4.5 hours to 8 minutes!!!!!

OMG man, was searching for 3 hrs how to extract javascript data w/o complicated rendering and your vid gave a 3 second solution. thank you so much man

Greetings from Brazil! Thank you! I just had to adjust some of the quote marks on the header (there were some 'chained' double quotes (like ""windows"")), making some of the header's strings be interpreted by python as code, not text. Just had to change inner double quotes for single quotes (e.g. "'windows'") and it worked perfectly!). Can't wait to try your other tutorials! Once more, thank your very much!

Tnx :) Went from 1 hour scraping with Selenium to 1 minute just getting the JSONs.

I was like "hm, okay, yeah" to "HOLY SHIT, THATS THE DOPEST SHIT I'VE EVER SEEN" I'm starting to get into this niche and I intend to learn more Python and SQL (you know, Data Analysis stuff/jobs) and I'm doing a project to scrape NBA statistics but there are always some errors and it ends up taking a long time. BUT THIS IS GOLD CONTENT, KEEP IT UP

nice tips, it's always fun to poke around and look at what data webpages are using

Very, very interesting - I'm going to give this a go myself. Cheers for another great video John.

This is such useful content that shows how much value experience gives - thank you for the straightforward and realistic tutorial!

Wow I love this! I was able to do it!! I did have to adapt to my own situation but still. Thank you so much

Wow that ‘generate code’ feature is super useful. Thanks!

Thank you so much for this tutorial. It helped me a lot on my project. And i learn a lot of new things that i didnt know. Thank you!

Thank you, your videos has automated my job. All i need now is a AI cam of myself

I love you, John. You're awesome! Thanks for being unique and producing quality content.

Great content, very useful now that I am learning about this subject. You earned a new sub here

This video is really amazing I learned web scraping from your videos thanks

That was incredibly helpful and exactly what I needed today. Your presentation is very clear. Thank you!

Easy to understand and very neat & clean narration. Keep it up 🙂

Thank you so much - this is so insightful and educational. Really helped me understand so many things in so little time.

John Great Video...Thanks for taking the time to do this!!!

Nice tutorial. But one important thing you haven't mentioned is that most of such APIs usually have some sort of authorization (based on headers, referrer, token, key, whitelist, etc.).

Thank you for such a wonderful videos. I learned a lot from you. BTW your video quality and background are always very beautiful.

Wow! I just found a gem of a channel! Love your content!

John, a specific video about how to scrape React website would be nice. It uses a mix of html and JSON data on pages...just an idea. Keep up the good work loving it.

This was nearly exactly my job back in 2014/2015 for a giant e-com shoe company. Was always nice when you'd come across a brand that included their inventory count in their API. But yes selenium/watir all day lol

brillant, i start to do that and it's very effective. Good chanel and good job. Thank you John

This is seriously high level content right here

Thank you! It provided a new way of thinking at the issue of collecting data. 🙏

Thank you for those videos. They're extremely helpful. Keep up the good work! 🙂

Wow, I think this 1 tutorial will do the most to up level my scraping than I could have ever imagined. Bye bye selenium.

Instead of looping over the results again, you can use res.extend(products)

Lol, I hadn't thought of a possibility to get an api like that until now haha ​​thanks a lot!

This single-handedly cut the running time of my program from literal hours to a couple of minutes, cannot thank you enough!

really nice juicy piece of knowledge. this XRH tab changes the game hopefully however there arise an issue of how to tackle the cookie-expire problem and if the api needs JWT token, private key

Thanks John! You are a lifesaver sir!

Instead of looping over the list and doing an append of each individual item, you can do list().extend(list()) which extends the list with the new list. The result of this is 1 list of dictionaries (basically an identical result to how you did it) but with less and cleaner code.

This has saved me hours today. Genuinely, thank you. 🙇‍♂️

It worked like charm! I really needed this. Thanks

Quite interesting! Thank you so much for showing such nice tricks, gonna get familiar with Insomnia.

Thanks a lot, very interesting video, i learned so many things that i didn't know. I will come back for sure!

You made/make stepping into scraping and developing, easy and fun . Thanks for sharing !!

John, you make scraping interesting and motivating simultaneously. Good that I found your channel P. S. I lost myself it at 0:10😂

Thank you soooo much.....I needed THIS!

Man, this is gold. Thanks for sharing!

Great video! Thanks so much for sharing! I think you should consider some academic research program (if you havent't already). I am sure you would do an amazing work. Congrats and thanks again!

Such a nice vid, if there was a VPN add, I didn't even notice it!

thank you for the information that you have explained, this is very helpful for the research I am doing

This feels illegal. And I love it!

Hi there, I found your channel where each and every video delicately made for web scrapping and automation which helps me a lot as work with web scraping and web automation. I have a request, if possible then please make python data post methods on Stateful api v1 and how to mimic cookies and session to get the job done. Thank you.

Hi John. Amazing content as always. Do you think I can skip learning scrappy for now? Can I do most of the scraping tasks just by using BS and request html?

This is amazing! So many things I didn’t know.

Complete rookie here. I’m trying to understand scraping to help access my lap times from the mylaps api utilizing my own interface. This is intimidating for a novice like me.

Thank you for making this awesome tutorial.

DevTools is a swiss knife for this kind of reverse engineering! It never gets old for me.

Hi John, I loved the video so much that I had to join Patreon to subscribe there to you. Thanks!

Nice video -- perfect level of detail.

My master in data extraction.

Well it sure is alot better then what API pulling I had to do in my last job it was nothing short of a nightmare.

Huge thanks this video was a game changer for me!

I used to do this in PHP without knowing better. Nice to see how it's done in python.

This is amazing and will help me alot. Thank you!!

Clear and helpful as usual. Thanks a lot!!

This is very useful trick John. 💖

This works in a lot of cases were the API is open. However, in cases like Social Media Platforms were you have to have an account to access the API or a Wordpress Websites were the API is turned off it wont work. The best approach in these situations, is really just to use Selenium or anything close and try to crawl the pages with a delay.

John brother your content is Amazing..........Pls improve audio quality...Respect from INDIA.

Great content. Thanks for this video

I was using this too and burp suite to have more control in some times

ive experienced being blocked when scraping a web site, but can i be blocked when i scrape through api? should i use sleep like when scraping an actual website? keep up the good work, not many wholesome web scrape content out there

Thank you so much for the tutorial. I have a question, how to get a Authentication value that include the header, can I do automatically and without selenium? In this moment, I get it manually in the network tab, further, the authentication value expire after of a time.

Wow, just did not know something like that existed. Thanks :)

Do you have any tips to get, for example, your own instagram followers? When I copy the cURL request and paste it in Insomnia, I don't get the same JSON as the browser, but instead I get some HTML which previews the Instagram logo. I assume it has to do with some authentication but I have know idea how to fix it.

Great information Sir, really helpful.

Thanks for this - and other - videos, John. Super helpful! Regarding the cookie expiring, can you suggest a way to use playwright to programmatically generate the cookie used on the API request? I am assuming that cookie isn’t the same as the cookie used for the request of the html but maybe that’s wrong?

Thanks so much. This makes things a lot easier

This is gold man, thank you! Just WOW.