Web Scraping with Beautiful Soup - Make Databases from Scratch 

But of course, Ian! 😃 I always appreciate your input, especially when something can be done more efficiently! Thanks for taking the time and sharing your solution 😊

@@PythonSimplified thank you

Thank you so much Jazna, glad I could help! 😁

You're welcome, I'm glad you liked this tutorial! 😀

It's not memorization, it is knowing how computers work. If you fully understand what is happening in each line written, you will remember the syntax.

Thank you so much Chiranjeeb! 😁 I wouldn't be so sure about the "allowed" part though, as there's nothing you can do with web scraping that can't be done manually with your keyboard and mouse! If something is not meant to be copied - you won't have the option to interact with that particular element, it's quite easy to do with simple HTML/CSS! But again - as long as you can do it with your mouse and keyboard - why would web scraping be any different? To support my claim, if web scraping actually got anybody in trouble - I would be in prison a long time ago along with all the cyber security experts, marketing advisors and software testing individuals. There is nothing illegal about automation and scraping if you're using it for educational or personal purposes! When you try to distribute this information, however, this is where you might get in trouble with copyrights - but as long as you keep the usage within the "fair use" guidelines - you don't even need to ask permission from the original owner to use his content. This allows us to freely use the internet without worrying about downloading pictures or coping text/code that was created by others 😉 In terms of social media platforms though - you are correct! The terms of service of some platforms may "not-allow" you to crawl or scrape - but they can't get you into legal trouble no matter how hard they try!!! They sure can delete your account or restrict it - but this is why I always recommend using a dummy account instead of your personal account whenever you're running bots 😁 If they delete your account - just open a new one and keep scraping like a boss! 🤪

@@PythonSimplified What you are saying makes sense. As long as you don't distribute you are good to go. Also websites have a robots.txt file. I think that has more info on which sub-domains might be restricted.

@@chiranjeebroychowdhury7759 yup! Or you can also include a robots meta tag inside the section of a webpage to prevent some of the scraping. It's great to combine with the robots.txt! With that said - you can always bypass it with with fancy Python commands or services like ScrapingBee that use proxies. Alternatively, you can even use Selenium to bypass these restrictions as it generates a browser window, which makes it very hard to distinguish between a legitimate user and a bot 😃 Headless browsers are a bit more problematic from that point of view...

@@PythonSimplified Yeah I am going to watch your Selenium videos this week. Never really got around to learning it. It's not really much used in academics. I am actually a CS teacher so I never felt the need of learning Selenium because it's not in many curriculums. But I do realize now that it's actually pretty interesting!! And I am so glad to see that you interact with commenters so promptly! Maybe you could consider having a Discord server or something similar.

@@PythonSimplified Hello. Anytime you browse a webpage you are downloading content. Anything you see on a webpage is currently on your computer. So, you don't even need to specifically request that content is downloaded as it is downloaded by default. If downloading content was illegal, ever user of the Internet would be a criminal. The issue comes when you use someone else's content for profit. I once found a picture of a squirrel on the net that I used in a website I was creating for a pest control company. Several months later the website owner received a letter from a law firm asking for damages. FOR A PICTURE OF A SQUIRREL!!!!! It was very unethical as the there were no damages. The picture was taken by someone on vacation and they didn't suffer any losses by its use on the website. We ignored the letter and that was the end of it.

I've been a huge fan of BeautifulSoup for a long time, and today I found out it actually has a "daughter" library named Mechanical Soup, which offers very similar functionality as Selenium. I thought I should mention it, as I'm still in awe with how simple it is! :)

Glad I could help! 😁😁😁

Thank you so much Khayam! I'll definitely check out ParseHub, I'm a big fan of easier methods! Thank you for letting me know :)

Thank you, I'm glad you liked it! 😃

You're welcome, enjoy! 😃

Thank you!! I'm really glad I was able to help! :)

Thank you! :)

Thank you so much The SouthSideDev, welcome aboard! 😃

You're absolutely welcome, enjoy! :)

Thank you! :)

Thank you so much!! :D I'm glad you liked the video and I really appreciate your lovely feedback! :)

@@PythonSimplified You're welcome. Good work must be appreciated. I came here from the FB Python group.

@@digigoliath that's awesome! I'm there all the time 😀

Hi Iftakhar, I'm currently working on a video of pulling data out of Instagram, which is based on React. Once I reach satisfying results, I'll definitely let you know. The MySQL part, however, is not something I'm planning on getting into in the near future, it's a bit more complicated than what I can teach at the moment, my apologies on that. Try asking in Python groups on FB, there's lots of support among the Python community and it might be something that other users will find helpful.

Your best bet would probably be Selenium, here's a link to the documentation: selenium-python.readthedocs.io/ It all depends on the exact web language you're looking to scrape, every module deals with different languages.

Hi Roshan, thank you! I've deleted most of my social media accounts due to privacy concerns, however, I'm still on Linkedin: linkedin.com/in/mariyasha888/ and I still didn't delete my Instagram account, I might use it in the future (or I might move to the middle of the woods and delete it too 🤣): instagram.com/mariyasha888/ I'm working on an official Python Simplified website, where we can communicate just like on social media, but with no sinister use of our data by Tech giants 😁 I'll keep you posted!

You can do this using MechanicalSoup

Hi Dury! I'm not using PyCharm so I'm not sure if I'm the best person to ask... but from what I've seen on Stack Overflow, PyCharm might have an issue with the bs4 library. May I suggest you use a different library altogether? Mechanical Soup is based on Beautiful Soup, but it's way more convenient and can do so much more! :D Here's a link to the documentation: mechanicalsoup.readthedocs.io/en/stable Or you can check out my video on Mechanical Soup: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-drDdb1MBBfI.html Let me know if it worked out! :)

You can either go for the innerHTML or text attribute, but if you're scraping a table - there's no point using Beautiful Soup, you can do it much quicker with Pandas. I have a super simple tutorial on it, you can check it out here 😃: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-oF-EMiPZQGA.html

Nope :) Unfortunately I'm not even familiar with the brand/channel... anything cool you recommend to check out?

Thank you! :)

@@PythonSimplified He was talking about the soup

@@aristideregal oh... ok 😢

@@PythonSimplified Just kidding. Your videos are amazing. A bit too hard to understand for me. But I am putting in the work to level up!

Bruuuuuuh