Intro To Web Crawlers & Scraping With Scrapy 

Right now I'm learning Bootstrap from your udemy bootstrap course. Man, it's amazing! it's not just videos or slides, it has a very comprehensive code examples which accompanies whatever Brad says in the video. Brad did a hell of a job in this course! It's wonderful! I highly recommend it. I've not tried other brad's udemy courses, but if they are as half good as his bootstrap course, I'm sold! This man is a god among us! Love you brad, the great instructor and the awesome family man! God bless you and your family.

Awesome video. I never thought I'd learn this much in 30 mins. Every second of video is full of useful information. Thank you so much

Thank you so much brad. I purchased the Django course on udemy. Awesome content. Congratulations. U will soon reach 1M subscribers. Wow.

Let's scrape ... the scraping blog! I had a good laugh. Your courses are amazing and every now and then we get a good laugh. Keep up the excellent work.

I did a little web scraping a while back-- this video is very timely because I was going to get back to it!! I needed a refresher, thank you!!!

Love your series! Thank you always!

great lesson. After doing some webscraping with selenium, this finally made a lot of sense because I was lost a month ago

thank you so much. My first time with Scrapy and you've been really clear. Great video. Tranks mate :)

This video couldn't have come at a better time... Thanks a bunch Brad... God bless

Thank you very much for this tutorial! It's nice, short and crisp!

for code 'page = response.url.split('/')[-1]'. I thought it should be page = response.url.split('/')[-2] and it works for me. But I donot know why it works with 'page = response.url.split('/')[-1]' in the vedio.

I have use scrapy for many web crawling and web scraping projects. However, I still found this tutorial very handy.

It's a little out of my league since I am only a beginner coder but it was utterly fascinating! Thank you very much!

I was just looking what a crawler is some hours ago. Now logging in to see this uploaded an hour ago! Are you reading the minds of your subscribers? :)

man after watching this video and executing this video in just one morning I managed to crawl an entire website in seconds. Thank you!!!!

I was following along with a different site I needed to scap all made sense still always grateful Brad

Such an awesome tutorial, sir!

Ridiculously awesome video! Def an amazing teaching and great start to web scraping with scrapy. Dope Stuff!

Your videos always great. A lot of other coding vids built on python talk about simple math for 8 hours and I learn nothing.

Thank you for the instructions, I like how the last minutes made things clear for me...

Very good, please keep doing this tutorial series :)

Great video, both compressive and concise!

thanks for your comprehensive description. i think this is good as start point.

Love your tutorial man. Thank you. With scrapy can we scrape millions of data with sequenced/scheduled interval to not get blacklisted and keep updating out file?

Awesome! Thank you Brad!

Excellent tutorial, as usual. Kudos!

Great Video simple explanation Thank you

Very clear explanations :-) Thanks a lot !

This is sooooo cool! Thanks a lot Brad!

I dont search for this but i am kind of like to watch this so thank you

Like => Add to Watch later => Thanks, Brad. :)

Great tutorial, the copy XPath from the browser was very handy

I love me some scraping but I did it with puppeteer and something else for work. My custom API did get blocked a few months later though...

Bro ur the best, I hate my life but these vids help make it better. I do IT and dev because I like it and because I don't have anything/anyone else for me. Thenk u for helping me learn, Been into python and C# lately, as I re visit JS, it only strengthens my skills, after thinking in New paradigms

This is a great Tutorial for crawling data

Nice and I know you've said there's lots more you could do with this, but one obvious improvement you could make to this is to collect an array or a set of URLs as you go to ensure you don't crawl the same page more than once - as I think that's what this code might end up doing, as it is right now. Right?

Your video is best, thank you help me a lot!

Thank you for sharing the knowledge.

Hey brad m still waiting for your new vanilla javascript course can you tell us when it will available in udemy???

This is great. Any plans for a Python video that calls an external API and fills models?

thx for this one, helped me a lot

wish you did a whole series on this

Very helpful, thank you.

very useful video super educative and clear

Hi Brad, thanks for the video. Is Scrapy also able to handle SPAs and specifically with content that is dynamically generated with Javascript?

Hey man, pls do a course on setting up a bespoke MVC system from scratch with express server , node etc.. going over the MVC fundamentals etc.

This is great content @Brad, would it be possible for you to explain some basic topics of SEO, I feel as engineers we often lack those skills, right now I am going through the pain. Again very grateful for every piece of content you put out there.

how are you know my thought. i looking for web scrapper and you make a tutorial with this ? are you an alien brad

Can we also take the user input for the url to scrape in scrapy?

Thank you!

Hello brad ! Please could you tell me when would yould you share the front end course for the devBootcamp backend on udemy?

Glad to be here

Thank you.. Great video.

What about pages that are secured with middleware can you scrap them aswell?

you're really doing good job ... keep it up buddy... joey says

In 8:04, why not use an f-string instead of the old percent sign way?

Can you please explain why you used yield on lines 13 and 21 for final version of code? Does this mean parse is generator function in this case? How does this work under the hood?

Nice video man ! which extension do you use to see scrapy help in vscode ?

so weird that i just started looking at scrapy this morning and boom... this vid drops. question - i cant seem to get vs studio to launch the debugger for a scrapy file. any secrets? thx

Lovely project!!

You are literally the best

Can you do a video about *unit testing* ? Please

how do i select all texts from for example "a class="new-class"? i dont want the text from other classes

Hi, Please advice me on how to improve / speed up the scrapy process

I haven't found any better videos for data structure & algorithm. If you know something please make a vid about it.

Thanks for this

What is the setup on your developer tools on chrome?

Great tutorial. when I import scrapy in the spider.py file I get a 'Unable to import 'scrapy'' on VS code, is there something im doing wrong?

Can you make a scrapping tutorial in js? There, maybe, so many persons who are looking for web scrapping tutorials in javascript.

In vs code how do you execute the python code in the terminal. Like when he starts the for loop?

Thanks for your explanation......but how can do that (crawl ) for my website which built with WP and js ? if some one help me in that

Hah I was just watching Scrappy Coco.

thank you one question why is necessary to create a virtual env?

I may be mistaken but I believe there is already a default method named "parse" that is overwritten here. Nothing wrong with overwriting it but it could cause unexpected behavior for someone that doesn't know.

How about if I have multiple keywords, for instance, “123”, “apple”, orange” or even with date time, can I use these before crawling it?

Next video: How to overcome captcha with Scrapy :)

do u have its course ?? or playlist where are other scrapy videos

great tutorial, but Im having trouble following along. filename='posts-%s.html' % page fails to number the pages so i just get post-.html and it overwrites itself for page 2, i assume. also tried filename = 'posts-{}.html'.format(page) with no joy.

you're the man!!

I see Brad's Video I click it even though I don't know what's going on :P . Like it anyways.

Wow! It would be great if you make JIRA course and Agile development, Love all your courses here in udemy, keep going sir.

Great video

Quick question, using xpath insead of css when generating with yeild creates in json file differently, I mean it puts all the titles first, then the dates and so on. It's a there a different sintax that I need to use?

So it ends the text as it bumps into an apostrophe in regex? Congrats to the almost 1m subscribers!

Hello! I want to do some web scraping but to find info on a certain thing. So normally, I would use a search engine to find the urls then from there, find the data I need. How would I automate the process of obtaining the URLS? The websites are pretty much the same ( I only really end up using 4 or 5 five websites with the data being a specific spot on the site). I would really appreciate any suggestions! Web scraping is such a good tool, but I need to automate the URL gathering process to accompany the Web scraping

Scrapping with js, just in the perfect time for me

Is there any upcoming course for Vue with TypeScript?

can u pick up viruses from websites while u scrap?

I have 2 dozens of sites with jobs in europe. I would like to crawl and scrapp several data sets from it. Is there a way to do this in a generic matter to get it all at once ?

What if the Website is heavy on JS? and how to manage the robot.txt that explicitly disallows Scrapy? :/

genius my friend.

Hello Brad, When 20 vanilla Projects course will release.. Waiting for that.

source venv/bin/activate messes my terminal design. How can I make the venv text appear in a different place (Edit:) Figured it out. check your activate script.

Nice video!

great very nice

Do we need to know Pyhton to scrape the data ??

Also tried a scraping with a node app. I don't know why but the performance was really different from this Scrapy.

What about Beautiful Soup

4:38 When I type 'import scrapy' I get the message 'unresolved import 'scrapy'Python(unresolved-import)' I am using vscode

Is it possible to code this normally like in pycharm or sublime without using a virtual environment?