Scrape Twitter with 5 Lines of Code

Подписаться 170 тыс.

Просмотров 65 тыс.

50% 1

In this video I show how you can scrape twitter data using the python library snscrape to easily pull millions of historic tweets and save them off on your computer. Use this to create data for analysis or just archive off your tweets quick and easy.
snscrape github: github.com/JustAnotherArchivi...
Timeline:
00:00 Intro
01:16 Twitter Scraper
03:37 Loop and save bulk
05:35 Add a Progress bar
Follow me on twitch for live coding streams: / medallionstallion_
My other videos:
Speed Up Your Pandas Code: • Make Your Pandas Code ...
Speed up Pandas Code: • Make Your Pandas Code ...
Intro to Pandas video: • A Gentle Introduction ...
Exploratory Data Analysis Video: • Exploratory Data Analy...
Working with Audio data in Python: • Audio Data Processing ...
Efficient Pandas Dataframes: • Speed Up Your Pandas D...
* RU-vid: youtube.com/@robmulla?sub_con...
* Discord: / discord
* Twitch: / medallionstallion_
* Twitter: / rob_mulla
* Kaggle: www.kaggle.com/robikscube
#webscraping #python #twitter

Наука

Опубликовано:

13 июн 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 244

@ioannisavgeros167 7 месяцев назад

Unfortunately, does not work anymore after changes. Returns 404 constantly.

@robmulla Год назад

Edit: It looks like this package is no longer working for some people (Thanks Elon) If you have issues here are some tips: From the SNScrape github page: ``` If you discover an issue with snscrape, please report it at github.com/JustAnotherArchivist/snscrape/issues. If you use the CLI, please run snscrape with -vv and include the log output in the issue. If you use snscrape as a module, please enable debug-level logging using import logging; logging.basicConfig(level = logging.DEBUG) (before using snscrape at all) and include the log output in the issue. ```

@chaymabarkaoui6506 Год назад

even I install the last version, it still won't work

@AbdulRehman-nu2pb Год назад

Yes it's not working 😭😭

@Kemmydai Год назад

Presently doing my MSc project and this video just pushed me forward so much! Thank youuuuu

@robmulla Год назад

Really glad to hear that.

@mulberryJamz Год назад

Hi Rob, I'm new to data science and I was wondering if you can make a video that shares the basics to webscraping from webpages. More specifically how to go about navigating and reading the parsed HTML texts of data to locate the strings we want like say content specific to a table or reviews on a webpages. Thanks for posting such easy to digest videos and sharing valuable tips to do things in a more efficient way!

@andygarcia6113 Год назад

You make it look so easy. Amazing vid!

@robmulla Год назад

Thank you! 😊 Hopefully you can put it to some good use.

@JohnFallot Год назад

oh man, this is incredible and exactly what I was looking for 🤩 Let us know when you decide to launch a Patreon!

@robmulla Год назад

Thanks man! No Patreon for now, just share with others who might also apprecaite it.

@philipomelo Год назад

Awesome. Simple and straight 👌

@robmulla Год назад

Thanks for watching!

@joaogollucci7877 Год назад

Amazing, will help me a lot with a project. Keep up whit the awesome content!!

@robmulla Год назад

Great to hear! Thanks for commenting!

@7382932 Год назад

Hi, thanks for the video super useful. I am quite new to all of this. What software are you using? And do you make those titles/write in each box separately? I am currently using PyCharm.

@rosiefish Год назад

He's using Jupyter Notebook

@zoe-mv6ji Год назад

I wish you were my lecturer/ tutor, I’m doing a conversion masters and the way you explain code is stopping me from having a breakdown 🤣🤣

@robmulla Год назад

Ha! Thanks, I apprecaite that. I try to make things easy to follow, glad to hear you're learning from it!

@zoe-mv6ji Год назад

@@robmulla you’re doing an amazing job at making it easy to follow. Thank you!

@nurulhanisahmuhammadhaasin885 Год назад

hi, thank you for the tutorial. do you know how to set the data for specified location or range?

@gustavolabrador6192 4 месяца назад

Hello, I wrote the code for the video, but I'm getting an error. It still runs as of this date due to the issue with the APIs that need to be paid for on Twitter.

@beethovennine Год назад

Another nice vid! Good job!

@robmulla Год назад

Glad you enjoyed! Share with a friend!

@beethovennine Год назад

@@robmulla Always do!

@stellamukuhi5646 8 месяцев назад

This is awesome, nice and straight to the point!

@ritamchatterjee3307 8 месяцев назад

and its not working

@omondifellow3120 Год назад

I'm geting 403 Forbidden error while trying to scrape tweets using the library. Does it means there is a restriction?

@ahmadjibrilhemdi7639 Год назад

thanks! your progress bar is awesome!

@robmulla Год назад

I appreciate it! I actually have a whole video on tqdm (the progress bar package) you should check out.

@hemalatha_karuppusamy 9 месяцев назад

Thanks for the knowledge sharing .... When i am trying the same i am getting the 404 error .. Can you help me with the error rectification..

@huyfe1347 Год назад

Hi Rob, thank you a lot for this video. I have two questions: 1. How to proceed if we want to scrap data on twitter with more queries like : #father; #mother; #baby; #foot etc.. Are we going to write one code for each query? 2. Did you make video on scrapping Facebook, Instagram, Mastodon, Reddit, Telegram like you did for twitter?

@devngecu 11 месяцев назад

use a for loop

@vtulin3097 7 месяцев назад

Hello,Would you mind if ask if you can demonstrate how to use a crawler to capture twitter space cations. If the audio file is very long (more than ten or twenty hours), how to quickly capture it?

@AminuteCrafts Год назад

Hi I am getting an error at line 6 for loop that Error retrieving , errors blocked 403 and request failed to giving up

@CaribouDataScience Год назад

Good stuff! Why did use a list first (data) rather than a data.frame?

@robmulla Год назад

Thanks! Appending to a list is much faster than appending to a dataframe so typically I prefer to just creat the dataframe after all the data is collected.

@uzairmughal9383 4 месяца назад

Instead of capturing tweets based on some hashtag, can we pull tweets from specific accounts? Like I have 5 accounts for whom I want the tweets data for a given time frame, is it doable from this method?

@sonnix31 Год назад

Sir, do you have a chat group where people ask questions? I followed your flight EDA, do you think predictions could be done with random forest ? Do you have something about this? (newbie here) . Thanks. BTW i like your videos.

@robmulla Год назад

Yes I do! Join the discord: discord.gg/HZszek7DQc

@ernestcholopray3545 Год назад

Excellent Rob. Would you happen to have tutorial sessions? if so kindly link me to it

@robmulla Год назад

Yes! Subscribe and hit the bell icon and you’ll be notified when I go live. Also follow me on twitch.

@ggggg-hr6en Год назад

Thank you so much. I have one question. When I use snscrape, I can only use "tweet.content, count, date, id, index, outlinks, outlinkess, tcountlinks, tcountlinksss, url". But I want to explore like counts, image url and so on. Would you tell me how to scrape them?

@robmulla Год назад

I believe likes are in there as well as the attached images. But you might want to check test it out. If snscrape doesn’t work you’ll need to get a Twitter api key.

@j.4880 9 месяцев назад

This is gold. 6 minutes and so much information given without any unnecessary fluff... I'm amazed. Thank you so much.

@ozgurbuldum6829 8 месяцев назад

Could you make it work? I guess the most recent reply is yours, I get " blocked (404) " error with Python 3.10.

@mahlatsembooi3375 8 месяцев назад

@@ozgurbuldum6829 yeah, blocked!

@mariodiaz2700 8 месяцев назад

@@ozgurbuldum6829 it does not work anymore right?

@elu1 Год назад

Very effective teaching! could you have a video for exploring data scrapped from twitter?

@robmulla Год назад

Thanks! I'd suggest checking out one of my live streams!

@Laufeyson_ Год назад

Thanks for the awesome video, are you planning on doing NLP project?

@robmulla Год назад

Thanks for watching! Have you seen my video on sentiment analysis? That has some NLP ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-QpzMWQvxXWk.html

@Azouni01 11 месяцев назад

Thank you for this wonderful tutorial. Q: How can I retrieve infos like tweet content , username ... ,for a specific tweet using its id or url?

@AIvarogarciaa Год назад

Hi guys, I have a problem with snscrape. The code works well but without any reason the process early stop at 188 tweets :/

@CarlosReyes-ku6ub Год назад

I'd really love to see you Forecasting Time Series using Transformers based Neural Nets.

@robmulla Год назад

I plan on making a video like this at some point. Only downside is that in my experience these models don't perform well unless the data is huge.

@CarlosReyes-ku6ub Год назад

@@robmulla You're right, but given the versatility of Transformers Architecture and the actual importance on the State of the Art it's worthy to know it, I personally haven't found good materials and since your videos are awesome I'd very much like to see it.

@khouloudsafi1445 8 месяцев назад

Thank you very much for the video,,, SNScrape used to work for me before but now it shows blocked 404, after some research I found that twitter now is blocking scraping without login..... any idea how to solve the problem ? Many thanx for the efforts

@magdieladames7233 8 месяцев назад

Interesting, I was just going to start using this. 😢

@nikolakalev4914 Год назад

It seems like the snscrape module doesn't work anymore, due to returns an error. Do you have any information?

@FRANKWHITE1996 Год назад

Just subscribed ❤

@robmulla Год назад

Thanks for subbing. Hope you like the videos.

@milesonme Год назад

Mate, how are u so good? What's your strategy? You code every single day? Or maybe you have many yrs of experience? What sort of courses or books do you study? How often do you read papers? I'm asking cz I learn so much jst thru recreating some of your projects and usually u have a very unique approach that I am used to, with very accurate results. I sure wanna be like you some day. Also could u keep the projects coming mate? I know there are tons of them on kaggle but it's quite frustrating to get stuck for us newbies a video explanation really helps.

@robmulla Год назад

Thanks for the kind words. I really don’t think of myself as being better than any other person with the same amount of experience. I think the main thing I try to do is learn constantly and be patient. Things take time to learn. Check out my video about how to get started in ML for a more detailed breakdown of the skills to learn. Good luck.

@yunes7305 Год назад

Hi Rob! How much time it takes per 100k tweet? Just to compare with my scraper built in another language which takes approximately 10 to 15min to scape each 100k tweets.

@robmulla Год назад

I think if you watch the end of the video the progress bar shows 35 tweets per second. So like 1 hour for 100k tweets?

@skez4774 3 месяца назад

Is your method still working today ?

@ketannanoti20 Год назад

I tried but showing error after "for" statement, Error like : blocked (403), Error retrieving and message "Bad Authentication data" code 215 what I do now? Actually I have some twitter link and from that link I want like , view , RT count .. can you help me.. ?? How I code

@DarthKidd Год назад

Hi Rob. Thanks for the video. Did twitter update their security? Using your exact code from the first 3 minutes of the video, I keep getting an error retrieving api requests. Any tips on how to get past that?

@eve9587 Год назад

Sameee. Did you work around it?

@DarthKidd Год назад

@@eve9587 I did not unfortunately. What about you?

@eve9587 Год назад

Dude use top=True It works now

@clintongomez Год назад

@@eve9587 Hi, I am having the same problem with snscrape recently. I am quite beginner on this. Can you explain where to apply top=true? Thanks in advance.

@robmulla Год назад

Looks to be an issue everyone is experiencing. You can follow the github issues here and they may update. Hopefully just a temporary issue... github.com/JustAnotherArchivist/snscrape/issues

@tarunanand4308 Год назад

Hey I am getting a 403 blocked error, I suppose the snscrape is going through API

@rishisharma6302 8 месяцев назад

Can i use this same code in vscode

@jp-sc5du Год назад

How do I get a specific tweet and retrieve the quotes and replies?

@iliecosminbiltan2639 Год назад

HI, Rob. I don't understand why i have this error: Errors: blocked (403), blocked (403), blocked (403), blocked (403) that's when i call scraper.get_items()

@sohaib5086 7 месяцев назад

same errror

@ahmedshalaby185 Год назад

Hello, Thanks for the nice tutorial, I just wonder if there is a way to scrape only tweets instead of reply and tweets

@robmulla Год назад

I’m not sure but it should be possible in the query method.

@Technomic-o Год назад

Amazing man

@robmulla Год назад

Thanks 🔥

@valerionetophdcandidate 9 месяцев назад

with this twitter x update is not working anymore? I tried to run the "for tweet in..." and it did not work "Error retrieving ...."

@robmulla 9 месяцев назад

Yea, I think Elon broke it. :(

@CaribouDataScience Год назад

Everything works (in VS Code notebook) except the progress bar isn't displaying? It runs without error. Here is the code: n_tweets = 5000 for i, tweet in tqdm(enumerate(scraper.get_items()),total=n_tweets): data = [ tweet.date, tweet.id, tweet.content, tweet.user.username, tweet.likeCount, tweet.retweetCount, ] tweets.append(data) if i >n_tweets: break

@robmulla Год назад

try importing the base tqdm instead of tqdm notebook: replace: from tqdm.notebook import tqdm with: from tqdm import tqdm

@CaribouDataScience Год назад

@@robmulla Thanks that did the trick!!!

@amiinofficial6272 3 месяца назад

how can i scrape specific tweets? pls answer me it is very important .

@abdelbaseermohamed1396 Год назад

The code is no longer working; it reports a problem of tweets blocking. Could you please check and help solve this issue?

@ayomideonikoyi Год назад

Yh noticed thusalso, something seems to be blocked.

@waleedmandourlecturer Год назад

Thank you for sharing this amazing tool and wonderful tutorial. Q: How can I retrieve all replies to a selection of tweets?

@robmulla Год назад

Thanks. I don’t know if that’s possible with snscrape. You might need the official Twitter api.

@tantomanontroppo8582 Год назад

hi rob, i think this api doesn't work anymore. i think twitter blocked it

@kanaparthyjaswanth9484 Год назад

The data seems to be truncated. could anyone please tell me how to deal with that ?

@mmfr3 9 месяцев назад

Does it still work with new API packages?

@CaribouDataScience Год назад

What would be helpful for me would be a video on how to connect Python to a SQL Server database.!!

@robmulla Год назад

It's really easy with pandas. pd.read_sql() - you just need to create the connection object which depends on your database type.

@ruirossi6241 Год назад

very nice. Thanks

@robmulla Год назад

Most welcome

@tommyrieder4562 Год назад

When I run the code the progress bar turns red and becomes "1001/?". Did I make a mistake in the code or can I fix this?

@scarfface Год назад

I think it's because that loop will actually gather 1002 results with n_tweets set at 1000. I think Rob was just trying to get ~1000, not 1000 exactly. Reason: enumerate starts from 0, and the loop is only broken once that number goes ABOVE n_tweets (e.g. 1001), so you end up with an extra result at both ends. Quick Fix: break the loop when i > (n_tweets - 2)

@robmulla Год назад

Good catch! This is a bug. It can be fixed by subtracting 2 from the number of tweets in the line before the break.

@user-jf3rf7jy9d Год назад

if I have the tweet's url or ID, can I get its content?

@haileyjoseph7522 Год назад

May I ask if I want to scrape from one specific user? How can I do that?

@robmulla Год назад

Great question. You can add any query search terms like you would in the twitter search bar. Here is a cheat sheet: media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf So to search a specific user you should be able to do something like "from:Rob_Mulla". Hope that helps.

@haileyjoseph7522 Год назад

@@robmulla Thanks for answering me! I'll definitely try that method 😁

@magicbrushes0 7 месяцев назад

it's not working for me I'm getting and error in retrieving the data 🤕🤕

@a.d.a.k2143 Год назад

how do i scrape the english tweets alone?

@andyyang8876 Год назад

is there a way to pull say, latest 50 tweets from a specific user?

@robmulla Год назад

Yes! I mentioned in a different response but you can use twitter's query to filter to specific users or dates. Check this cheat sheet: media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf So do something like "from:Rob_Mulla"... And setting your loop to only go through 50 would get you the latest tweets.

@lumineersgaming4322 Год назад

why the tweets extracted showing only half text? why not full text plz help

@robmulla Год назад

Oh no, I haven't experienced that. Have you checked the github issues to see if others have the same problem? github.com/JustAnotherArchivist/snscrape/issues

@aidanheffernan652 Год назад

This with tweepy makes for a very interesting reposting bot

@robmulla Год назад

Interesting idea.

@behdad0979 Год назад

Does Twitter block you if create a bot with and get tweets every day?

@robmulla Год назад

I’m not sure. Things are changing every day at Twitter. This package has been relatively unstable lately so you might want to check out their GitHub (linked in the description)

@techyoungster2713 7 месяцев назад

It is not working now days. Do you have any other alternatives?

@impac143 Год назад

Is it possible to scrape profile information and followers list? thank you

@robmulla Год назад

That’s a great question. There are ways to do this but I don’t believe snscrape can do it. You might need to apply for the Twitter api and use something like tweepy

@marioaguilar8735 10 месяцев назад

How can I solve the 404 request error that I get when looping through the scraper object? Any idea?

@robmulla 10 месяцев назад

I think they’ve blocked it unfortunately

@TheFJudge-yc1ts Год назад

an error 403 from twitter api occurs at get items

@ishangoinyambo6523 2 месяца назад

is this still valid with x though?

@huyfe1347 Год назад

how can we install snscrape?

@CaribouDataScience Год назад

What is the mark after the "for i ,"

@scarfface Год назад

The comma after the "i" is to separate out the two things enumerate returns (index, value) in to the index "i" and value "tweet". Does that make sense?

@robmulla Год назад

What Tim said! Thanks Tim!

@vedantpanchal2291 Месяц назад

big headstart in a project

@josephdickson5082 26 дней назад

does it still work? i doubt that. Was getting a 404 and 429 status codes last time i tried

@milesonme Год назад

Can u limit the tweets to only tweets in a certain language say English?

@robmulla Год назад

Yes, use something like "lang:en" in the query. Check out this cheatsheet on twitter queries. media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf

@louloumattinson994 10 месяцев назад

Hi Rob, I cannot seem to make it work, I am not sure if its something I am doing but it produces a 'Errors: blocked (404), blocked (404), blocked (404), blocked (404)' message and states 'failed, giving up', with a lot of links to twitter that show I don't have access. Do you have any suggestions? Thanks!

@MrChefMedz 9 месяцев назад

same issue

@yugpatel588 4 месяца назад

snscrape no longer works for twitter. Elon has changed the policies making it hard to scrape without using the official API.

@rishavbhardwaj8044 Год назад

Hi, just wanted to ask, we need twitter API access to do this scraping, right? Cause when I run the same code, it sends a request to twitter API website and then an error is displayed.

@khadijadar3156 11 месяцев назад

did you find any solution

@rishavbhardwaj8044 11 месяцев назад

@@khadijadar3156 yes, I can scrape tweets now. It was a very simple code.

@TemporaryForstudy 11 месяцев назад

@@rishavbhardwaj8044 can you tell me how you fixed that?

@jaiswalrohan 11 месяцев назад

tell us how!

@rishavbhardwaj8044 11 месяцев назад

@@TemporaryForstudy I was using this code, but as of 2 days ago, it stopped working: import snscrape.modules.twitter as sntwitter import pandas as pd limit = 1000 tweets_list = [] for tweet in sntwitter.TwitterSearchScraper("NestleIndia").get_items(): if len(tweets_list) == limit: break else: tweets_list.append([tweet.date, tweet.content, tweet.user.username, tweet.user.displayname, tweet.user.description, tweet.replyCount, tweet.retweetCount, tweet.likeCount, tweet.lang]) df = pd.DataFrame(tweets_list, columns=['Date', 'Content', 'Username', 'Description', 'Language']) # Save the DataFrame to a CSV file df.to_csv('tweets_nestle.csv', index=False) print("Data saved to tweets.csv")

@hangyuli3328 Год назад

Sir, is it legal to scrape data on twitter without using twitter api now? I heard twitter shut down their free api.

@robmulla Год назад

I don’t know. I’m not a lawyer but you should be ok.

@hangyuli3328 Год назад

@@robmulla thank you

@EmailXpertise Год назад

what software do you use for write the code ?

@OfficialB23 Год назад

it's look like jupyterlab

@robmulla Год назад

Yes, jupyterlab - I have a whole video about it!

@user-dy9yo4yq9s 8 месяцев назад

My code is not working

@mohamed-muneer Год назад

I have seen that you use jupyter lab in dark mode, how to activate dark mode?

@robmulla Год назад

Solarized dark theme! Check out my tutorial video where I explain how: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-5pf0_bpNbkw.html

@Ketan3101 Год назад

How can I get follower count of different users using this??

@robmulla Год назад

I don’t think so. The Twitter api should give this easy enough though.

@tonymok7752 Год назад

Why your jupyter notebook interface looks different than mine?

@robmulla Год назад

I got into detail about my jupyter setup towards the end of this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-5pf0_bpNbkw.html - Hope that helps. In short: jupyterlab with solarized dark theme.

@tonymok7752 Год назад

@@robmulla Thanks. Imma do it with my setup

@patrickonodje1428 Год назад

Awesome

@robmulla Год назад

Thanks Patrick!

@Pscytlk Год назад

how do I get only people account and also their follower and following count?

@robmulla Год назад

Not sure if you can do this with snscrape, but with tweepy you could.

@Pscytlk Год назад

@@robmulla thank you sir!

@aliathar891 Год назад

can we access tweets older than 7 days?

@robmulla Год назад

At least you used to be able to

@ananth4595 2 месяца назад

the coding is easy but currently, it's not working. I tried many times 🤕

@bishnudas8638 25 дней назад

Hi Rob - I tried but it is not working anymore.

@siddharthmahendran8186 Год назад

Will I be able to scrape data using google collab?

@robmulla Год назад

You should be able to!

@evansyaran233 11 месяцев назад

sure

@AbhishekKumar-wf9ey Год назад

please help with the 403 error if you can.

@robmulla Год назад

It’s possible they may have blocked this

@pedropablocalderon5012 Год назад

@@robmulla Fix it

@muchammadfahd-a1985 Год назад

i've try now but sntwitter cannot get any items

@robmulla Год назад

Oh no. Did Elon shut it down?!

@WetBet Год назад

What is the thing you’re using to run the script? I know nothing about coding so I’m confused

@amasoharnold4762 Год назад

Juypter notebook just google it and you’ll find it … or search anaconda It’s not the snake I promise…

@muchammadfahd-a1985 Год назад

i have an error "Unable to find guest token"

@robmulla Год назад

I'm not sure why that would be, works for me! Let me know if you get it working.

@aidanheffernan652 Год назад

came across this error you need to initialize the scraper object after before every run of .get_items()

@danielhall6759 Год назад

Does anyone know if this method still works today?

@jaiswalrohan 11 месяцев назад

nope

@startupschool4895 Год назад

What shall we do for reddit, Make one analysing WallStreetBets.

@robmulla Год назад

Great idea. Maybe I’ll make another video.

@muchammadfahd-a1985 Год назад

hi rob? are you have a new trick for scrapping? i need this for college. thank you for attention rob GBU.

@murtazahussain8485 Год назад

The request was blocked by the twitter.

@ipakolaf Год назад

brother can you make a video on followers scarping

@robmulla Год назад

Ohh. Great idea. I need to figure it out first. I think the Twitter api would handle that

@IshaqKhan010 Год назад

not working anymore

@jaiswalrohan 11 месяцев назад

ok people this aint working now dont waste your time here

@paulapareja7835 11 месяцев назад

Hi Rob, thank you very much for your video. I tried to scrape Twitter with snscrape but it seams like the new API version blocks snscrape. Could you do a new video to scrape it without the API? Is there any way to do it?

@jaiswalrohan 11 месяцев назад

thank god i saw your comment before wasting my time just like i did on tweepy😭 apis aren't working now! f$u elon