Learn Python - Web scraping a private API - Questions from the comments episode 2 

I am taking two of the most highly rated courses on udemy about scrape and they do not have half of your production, and teaching you are great. Success for the future!, Éxito.

Helloooo, thank you so so much for literallyy making a video about my comment. Learnt so so much about python and API requests. You are one of the best teachers in youtube period. This certainly gave me a head start in my project and I can wait to complete it ! However, there is this one issue I am facing. According to calculations for 1 million job listing, there should be = 1 000 000 / 30 = 33 333 pages, given 1 page has 30 listing. But whenever I cross the 332 page mark and hover into pages like 400/500 I get the following message, " {"message":"[query_phase_execution_exception] Result window is too large, from + size must be less than or equal to: [10000] but was [1000020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.","error":{"status":500}} " This is a major problem I am facing. Given the genius you are, I am sure you will come up with an idea to fix this lol. Thanks in advance loll !

Loved the enthusiasm when you were checking the website for data! This was a great course. Just what i needed. You got a new subscriber :)

Great production and knowledge. Made it look real tight. Also like the way you explained it all. Subscribed!

Hey!! yes, exporting to SQL would be a very nice thing to know

Amazing value. Wish you make vids like this all the time(make money with python)! Thank you!

Really good video on web scraping. Please do more of that kind of videos. If you could do the video of how to feed the data into a spl database that would be awesome - thx.

This was really good content! I was able for follow along on my system and got the same results.

Awesome.

Great man !! I will be using this in near future 😀😀

Great tutorial! Thanks a lot.

Thank you bro!

Great video, thank you

nice

Awesome video! Subscribed! Quick question for you though. On the scraping project I'm working on, when I go to copy the cURL bash into the converter as you did, mine has a cookies section as well as the headers, params, data, and python request code. What do you think that means about the site I'm scraping? Should I delete the cookies section of the conversion? Cheers, Joe

im high af and that intro made me laugh

great lesson !!! It is really good for slow learner. can someone tell me why the data df has only one row and one column?

Thank you for the great content! I'm wondering if there is any way to get this approach to not fail if there is javascript, or at least be accepted as a real and current browser. I'm aware that copying out the curl provides all the headers/user agents etc but some websites seem to still be able to tell that it is not a real browser, perhaps it is because javascript is not rendering properly and it gives it away? any thoughts would be much appreciated!

shouldn't lat be first and then lng??

Hey man ... what is the tool that you use to work with the data ? Edit: It's jupyter notebook ... I figured it out watching the first episode of "making money with python" .

Hi, it is Barrsido from Reddit. I'm having another problem with my code. Most of the game is done, but for some reason the 'bal' variable does not update so during the betting, results, and scoring phases, it messes up on the second run. I put it back into the codeshare. Please msg me on reddit if you see this.

Is there any way to send a request in order to find potential acceptable parameters? (Once you've already found an useful api curl)

One question it's may be related .. I do have user id and password for a website now I want to scrap the data from there ?? How to use this technique ( the one you showed in video) on scraping those data??? Probably if you will give some hints or direction that would be good if you will get time you may make a video may 😀😀 . Thanks 👍

can use while loop for the page scraping automatically

Please share the code.py file with us

Vote up for sqlite

May i have your email address sir? I need your guidance how to scrape webiste that required username password and entering captcha to login?