Web Scraping 101: A Million Dollar Project Idea 

Let me know any web scraping project ideas you have! Sign up for @BrightData here and get $15 in FREE credit! brdta.com/techwithtim.

I did similar projects in the past (in php long time ago with curl and tidy, now in python with bs4 or selenium). To avoid bot detection, you can use the tor network, detect captcha request, and simply reconnect tor to get a different IP. Of course, you still need to put random sleeps here and there. Also, you need to be prepared to update or rewrite your code any time the website changes. Nice video!

So this is basically an elaborate commercial

Just an anecdote, but when I first got into a data role I built a competitor price web scraper for my company. At the time it was supposed to be a show and tell for my boss in case python work came up in the future. It worked well enough for the business to utilize for beating the comp to vendors on price. We could keep tabs on them at a quick enough pace to adjust quotes and win. Didn't always work out but being able to say a python project netted three extra contracts is really cool

Great video! Thank you for talking about web scraping as an IT subset. What most people don't know is that it's extremely hard to build your career around it. I have been running my own web scraping software house for several years now and getting clients is more difficult than if you build apps, create websites, or do virtually anything else. People and companies simply don't have enough awareness what access to real time data can do for their businesses. Creating this awareness is actually the biggest part of my job. Writing code is just a formality afterwards.

Wow! I was literally thinking of doing the same thing, but really had no idea it was that hard or that some many people have already tried the same thing and failed. Before I even started facing the problems, you gave me the solutions for free... So thank you!

I totally agree, I've done a couple of web scraping scripts for some companies, and to get some specific data, however there are a lot of challenges. But in general the rate limiting for APIs and changes in DOM, or catchaps are the most common ones.

I'm trying to build as many projects as I can right now to upgrade my skills and cv. This project idea is really helpful to my resume, I'll try to do it using fastapi and mysql (if possible I'll try to track stock price). Truly appreciated for sharing this project, sir!

A wonderful video that we've used as a reference for our recent additions. Your sharing is highly appreciated!

Thank you! This is the replacement for the outwit browser I've been searching for!

Thumbs up if sponsored tools are not your content!

Great tool Tim, i look forward to playing around with this. I still have a lot to learn about the Data industry though, keep up the amazing work.

Hey Tim, you always have amazing content. Keep it up! Greetings from Italy!

I love web scrapping; web scrapping is really satisfying when you finish your project.

Thanks for this valuable content Tim. I found it helpful

Very helpful looking for data will check out bright. Thank you.

Need a whole series with videos like these

Tim, your videos look very professional. I see the gear/hardware you use. May I ask what software you use to record your screen and include a picture-in-picture of yourself? I did not see this info anywhere. Thanks in advance!

Sir i watch your all videos and they are very helpful thankyou to provide as like that informative content.....👌

Bright Data makes it easier to steal data from websites that try to protect themselves... Great job BrightData!

Thanks for inspiring me to learn programming, you're really cool. This is a very cool and interesting project. ⌨💪🧑‍💻

This is exactly what I was looking for - thank you!

That's awesome, keep going bro

thanks for the video. I have been thinking of making something similar. Even bought a domain for it.

Man u gave us so many valuable informations,we appreciate it❤

Great stuff Bro i was wondering how to do this for long time

Great video ❤ one thing I noticed that might be helpful on your code is comments when you’re walking through what each part does. That way anyone who uses it from your GitHub doesn’t have to reference your video each time to hear what it does. Just a small detail I noticed. Otherwise, fantastic idea and project!

Tim deserves more than 1.26 million subscribers. Fantastic job!

🎯 Key Takeaways for quick navigation: 00:00 💡 *Potential of Web Scraping for E-commerce* - Web scraping offers real-time data collection for industries like travel, e-commerce, healthcare, and real estate. - The example of a drop shipper using real-time information to gain a competitive edge. - Highlighting the multi-billion dollar industry potential and the impact on business decisions. 01:27 🤔 *Challenges and Need for Web Scraping* - Acknowledgment of challenges: companies actively blocking data scraping, API limitations, and outdated information. - Introduction to the solution: the necessity of building a web scraper to overcome obstacles. - Mention of community suggestions on using frameworks like playwright or selenium for the project. 02:30 🌐 *Bright Data as a Solution* - Introduction to Bright Data as a sponsor, providing a solution to bypass website blocks. - Explanation of Bright Data's scraping browser capabilities, including solving CAPTCHAs and rotating IP addresses. - Emphasis on scaling capabilities, running multiple instances simultaneously for efficient web scraping. 03:59 🛠️ *The Fully Functioning Project* - Demonstration of the developed web scraping project: a product search tool for e-commerce prices. - Overview of features: enabling/disabling product tracking, adding new products, and viewing individual prices. - Encouragement to explore the open-source project on GitHub for customization and extension. 07:02 ⚙️ *Project Architecture Overview* - Brief overview of the project's architecture: frontend (React), backend (Flask), scraper (Python with Playwright). - Explanation of the database setup using SQLite for simplicity. - Description of various API endpoints, including results submission and tracking product updates. 09:03 🧩 *Web Scraper Logic and Bright Data Integration* - Insight into the web scraper logic, dynamically allowing integration with different websites. - Demonstration of connecting to Bright Data's scraping browser for unblocked access. - Explanation of core logic: connecting to the browser, loading pages, and retrieving product information. 11:03 📅 *Scheduler for Automated Updates* - Introduction to the scheduler for automated updates: triggering the web scraper daily. - Explanation of the Windows batch file and process scheduler setup. - Mention of hitting the URL to update tracked products and the frequency of running the automation script. 12:33 📈 *Potential Project Extensions* - Suggestion for extending the project: building an alert system for notifying price changes. - Encouragement for users to explore and enhance the project, potentially incorporating an email or text alert system. - Recap and conclusion, urging viewers to like, subscribe, and find inspiration in the presented project idea. Made with HARPA AI

Please keep project ideas video coming 🙏

Super helpful, thanks

He is right, I work for tech consulting company. One of the client projects I work on is web scraping project that collects doctor disciplinary action data using python and they pay a lot for us to do that for them.

thank you my king this sounds so cool

20$ / GB is huge and you will hit that really fast lol

Thanks for sharing this project with us !!! 👍

Hey i watched some of your videos and i like what you do here ... i was thinking may be if you could do a tutorial on how to Develop an API with Python or JavaScript

Thanks for info

How many pages can you scrape using this approach before Amazon blocks your scraper, and what's the best approach to avoiding detection by Amazon and other sites?

very great vedio like every time

Good idea. If u r in 2013. That's plenty of services who are provided same possibilities

appreciated brother. I was actually building something very similar but yours is definitely better build. Do you recommend against doing this in Javascript ?

I am planning on working on something like this, but I am from africa and i dont have too much knowledge of these GDPR laws and so on, could you make a video on that please?

Amazing project scraping the web

Thanks Tim. Any plan for a detailed tutorial?

What language would be the best to learn to create an app? Java? C#? Which one should I choose? Also, I want to know the prospects of each language. Can you tell me?

This product would be a smash hit among Coupon Grandmas in Texas.

Great project

Hey, thanks for the great content! Guide me please, what is the road map for learning the tools to understand what you have done in this project and be able to do it on my own?

I've seen people exploiting online job portal sites like Upwork. They hire freelancers from the Philippines to scrap websites rather than building a program, and of course they *CHEAP AF* like we're talking about $2-3 per 100+ websites. Most of them charged $2-3 per hour which is absolutely disgusting.

I love this dude deeply this was the reason I started learning code❤ now some one is give me a repo

A simple Chrome extension build would have made this so much easier for you and it's completely transparent for bot blocking.

I did this back then with food delivery services for example burger it will display the cheapest delivery and price from a pool of available service.

I did something like this for a company I worked for. Setting up slowly changing dimensions tables in python is p hard when there aren’t primary keys for the data you’ve scraped.

I tought I had some good skills on web scraping, then you showed me I'm just a baby still. Loved the idea and the fact you gave it out for free

Legend!

Thanks for this video. How did you bypass Playwright default timeout? It sees that Playwright has a default timeout of 3000ms regardless of setting await in line 106 to 60000 or even Wait=commit

Hm-m-m... Interesting approach... I'll definitely will learn it later, since I've used only the simple Web Scraping solutions, and it doesn't work out sometimes... Finger up, Tim, for your work, anyway!!!

Thank you

Thanks

Mr Tim is automation with java a a good thing ? wich better java or python for automation

I'm definitely adding this one to my resume

Just wondering haven't played with the program yet. But does the program also record prices differences if items have multiple versions example Red rain coat vs the same rain coat but it's Blue and cost $1 more.

Can we run whole project without brightdata functionality for website which do not blocks or medium level security in it?

hello Tim, is it possible to make a video on how to download the newest version of chrome driver, been stucked on it for days... i wanna crai

The idea is cool(the frontend part) except the "pay us to run your code and provide you the data data your code got" part, but what if you would want to scrape like 100k distinct items? The frontend will become just a long scrollbar.

thank you

Would you offer a scaper in RapidApi or any other marketplace?

Hi Tim. Thank you for your videos. Recently I watched your video about Selenium and your site in the video was very slow. Is it because of the slow server or because of Python? I've read a couple of articles about web developing stacks today and Python was mentioned just once. Does it mean that for a web developing it's better to you somthing else except Python?

Hello Tim, i'v got problem mostly with style. Css even with the guide of chat gpt, advise me please

Isn't Web Scrapping Illegal? Some time ago I had the idea to do a project just like this one and I ended up giving it up because I thought I was illegal. I read it on the website's terms and conditions.

Tech With Tim, dude, big fan, suggestion, get a ton of sleep like a TON like 24 hours then eat and take a shower then a really cold shower, and then go get a professional picture made or just use your camera and then use that pic as your profile pic cause on your profile pic u look like you're going to pass out from too much hard work too many hours (which I understand being a techie myself) your eyes are giving you away. Love your stuff man, thank you tons, keep it up PLEASE, TKX!

bright data ok cool got it !

I want to create a search algorithm that can find files that contains a particular word or sentence and open it using python. I save passwords and other things to remind me using note pad but I want a way of typing a word or sentence or part of a sentence and have the document open and the word highlighted.

Refined marketing approach... Brilliantly done

Hey Tim, i've learned some basic python from youtube, practiced in codewars, currently approaching 1kyu, i really love coding, i want to take it seriously, i wanna be job ready, i've no guide or mentor, i am looking for some serious advice and guidelines, because just feeling like a radarless boat, i don’t know what to do next.....,please help, and can i help you anyway?

I'm trying to scrap documents from a website with multi layers, so I can use those files with a logical chat bot to reference what I'm looking for, please any help will be appreciated, thanks in advance.

Can you tell, what thing did you do to monetize through webscraping

this is so sickk ima mess with it rn

Hi, can you make a video on how to deploy your web scrapper app on a online VPS please?

Just the cost of Bright Data makes this not applicable. I want to make 10,000 requests a day at least. Anyone found a work around?

youtube videos these days are becoming ads

More uses: - job searches - combine with NLP to obtain live information regarding sentiment towards products or services

Great video. scrapers + llm apps are going to dominate data very soon (if they aren't already doing this)

Do you have a video of web scraping data using javascript and nodejs?

instead of using any third party library can we integrate scrapy in it

In order to run it every day for example you should host it an make a cron job for it ?

I can build a web scraper in Python selenium that spits data into a csv. But where could I go to learn to do something like this?

I'd like you to build a Encryption and decryption tool

this looks like django? can you talk about deploying it? idk why i had such trouble with it

good day tim, i just want to mention and ask permission that i will extend this base code. for now i made a private repo and soon on the right time i will make it public and will credit you. thanks

Thanks a bunch for this Tim. But why still using create react app tho?

Who is buying scraped data - companies would just build it ? or is the value proposition any community that wants/needs to know if data has changed and building some kind of tool to do that?

Zeus Proxy ensures anonymity and privacy while performing SEO tasks, enhancing security and reliability.

that's great

Don't forget the proxies while web-scraping 😄

Always love seeing your videos really helped me when I was starting out and still a great source for learning new things here and there. Also I know your go to tends to be python and that you have done content with SQL and APIs with Python but in terms of teaching new programmers especially those that come to work on my team the hardest concept to teach is just the overarching idea of how The web in my case that's usually an Angular Project with TS interacts with API (Endpoints etc) and how that API links to our SQL database etc basically just teaching how it all connects from an oversight perspective. I think a video like that would be helpful to newcomers obviously the individual pieces are necessary to know but how they work together is important to ^_^ Regardless Keep up the Great vids :)

Hello tim I have a problem because I asked for a code for the enter button (when you search in Google and when you press the enter button, it gives you the result) Thank you for helping me.

we watch you grow up

Would you consider a full length setup and code tutorial for this project or a similar one?

Does anyone has a good tracking tool for specific website or search enginges for example if any tender is realsed?