Django + Celery + Selenium to Scrape Anything with Python

Подписаться 348 тыс.

Просмотров 61 тыс.

50% 1

🚀 Sign up for Bright Data Now brdta.com/justin
Selenium with Bright Data can scrape nearly any website regardless of Captcha or Login required. In this series, you are going to use Django and Celery to ensure you can scrape on your schedule and store the data with excellent structure. We will be tracking Amazon product price fluctuations.
Topics Covered:
⦿ Integrating Django with Celery for scheduled and background task processing
⦿ First-hand experience with Selenium in Jupyter for live browser automation
⦿ Advanced proxy scraping techniques using Bright Data and Selenium including handling complex challenges like Amazon Captchas
⦿ Building helper functions to streamline your scraping module
⦿ Combining Django and Jupyter for an enhanced data-driven development experience
⦿ Data parsing with BeautifulSoup4 to effectively find and extract the data you need
⦿ Efficient data storage using Django models
⦿ Offloading heavy scraping tasks with Celery for improved application performance
⦿ Automating and scheduling scraping tasks within the Django admin using Celery
⦿ Plus, much more in-depth knowledge and practical tips!
Recommended to Watch
⦿ Try Django 3.2 (or similar): • Try Django 3.2 - Pytho...
⦿ 30 Days of Python (up to about day 15): • 30 Days of Python ~ Py...
⦿ Some HTML / CSS Experience is ideal
Resources
⦿ Code: github.com/codingforentrepren...
⦿ Subscribe: cfe.sh/youtube
⦿ Django: djangoproject.com
⦿ Bright Data: brdta.com/justin
🚀 Next up? Web Scraping with Python + AI our Free Udemy Course: www.udemy.com/course/smarter-...
00:00:00 - Welcome
00:02:23 - Tools
00:06:07 - Rapid Fire Django + Celery Integration
00:19:06 - Setup Project
00:21:23 - First Look at Selenium with Jupyter
00:25:08 - Proxy Scraping with Bright Data + Selenium
00:32:08 - Helper Functions for Bright Data Module
00:35:55 - Django + Jupyter Integration
00:39:00 - Parsing Data with BeautifulSoup4
00:52:02 - Amazon Captcha + Prepare Parser Helper Functions
00:58:30 - Saved Scraped Data with Django Models
01:10:51 - Celery Task to Offload Scrape-to-Save
01:22:29 - Schedule Scraping Events with Celery in the Django Admin
01:33:16 - Thank you and next steps

Опубликовано:

1 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 26

@CodingEntrepreneurs 4 месяца назад

Thanks for watching! Here's the code: github.com/codingforentrepreneurs/Web-Scraping-with-Django-Celery Grab the free Udemy course: www.udemy.com/course/smarter-web-scraping-with-python-ai/ 00:00:00 - Welcome 00:02:23 - Tools 00:06:07 - Rapid Fire Django + Celery Integration 00:19:06 - Setup Project 00:21:23 - First Look at Selenium with Jupyter 00:25:08 - Proxy Scraping with Bright Data + Selenium 00:32:08 - Helper Functions for Bright Data Module 00:35:55 - Django + Jupyter Integration 00:39:00 - Parsing Data with BeautifulSoup4 00:52:02 - Amazon Captcha + Prepare Parser Helper Functions 00:58:30 - Saved Scraped Data with Django Models 01:10:51 - Celery Task to Offload Scrape-to-Save 01:22:29 - Schedule Scraping Events with Celery in the Django Admin 01:33:16 - Thank you and next steps

@amitavasengupta5580 4 месяца назад

Awesome!!!!!!!!! Was waiting for a celery video from the Master himself!!!! Had a tough time with celery now it's a time for a celebration

@CodingEntrepreneurs 4 месяца назад

You got this! 👍

@banka990 4 месяца назад

You are my only hope to make a tutorial on integration of Django with Next.js and demonstrate different types of rendering - SSR, CSR and SSG. Please

@diogos7233 4 месяца назад

I hope he sees your comment. I wish to learn this too.

@yssplays2620 8 дней назад

The best youtuber i ever seen

@edwardmike7523 Месяц назад

I learnt alot, thanks so much. But why did you use Django, if you need its model you can use Pydantic, SQL-alchemy, PostgreSQL

@kaanatsever2333 4 месяца назад

If you say we are scrapin, then we are scrapin

@CodingEntrepreneurs 4 месяца назад

😂😂

@abolfazldaneshvar1441 4 месяца назад

What a good tutorial! Keep going man

@CodingEntrepreneurs 4 месяца назад

Thank you!

@dgiri2333 4 месяца назад

1.Sir do text to (sql or ORMS) ask our questions and get result that way Course can you plan sir like Langchain sql to openai /2.geodjango related ask questions and get answers from our database

@aashayamballi 4 месяца назад

thank you!

@gamingLiberationFrontOfficial 4 месяца назад

sir can you please one on Nextjs Django and Docker

@MehrdadMahmoodpoor 2 месяца назад

great, can you tell me the name of font that you are using in vscode?

@not_amanullah 4 месяца назад

thanks ❤

@philtoa334 4 месяца назад

Nice.

@alexdin1565 4 месяца назад

how much Bright Data costs for 1000 scraped URL?

@adirnoyman2231 2 месяца назад

I’m trying to ping the redis-cli but getting this error: “redis-cli zsh command not found”. The venv is active and Docker is running the Redis image. What can be the problem?

@CodingEntrepreneurs 2 месяца назад

You need to install the redis-cli on your machine. It’s different from what you mentioned. The Python redis client should connect to your docker instance of redis without the redis-cli

@alexdin1565 4 месяца назад

at the right time because I got blocked a lot hhhhhh please can you make a video on how we can deploy Django with docker to server. I'm one of your udemy Django core if you can update the content

@CodingEntrepreneurs 4 месяца назад

Thanks for the suggestions. I do need to update Django Core. Thanks for the feedback