Topic modeling with R and tidy data principles 

HA you are so welcome! I'm really glad these resources are helpful. 👍

I've also got this blog/screencast that demonstrates how to use STM: juliasilge.com/blog/spice-girls/

THANK YOU so much!!!

@Tereza S look on my video on sentiment analysis on many documents ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-rU97L9Tu7Dg.html

She's written a book about it bro lol Here's the link www.tidytextmining.com/

I don't think you need to filter out synonyms before implementing topic modeling, because that is one of the things topic modeling is doing, during the modeling process, finding the latent topics. Related, you might want to even consider whether stemming is useful for your domain space: transacl.org/ojs/index.php/tacl/article/view/868 I have had consistent, excellent results with STM, which is one of the reasons I recommend it to folks. LDA models are based on the Dirichlet distribution (if you draw a sample from a Dirichlet distribution, you get a positive vector that sums to one); these models are based on priors over topics/words, then you solve for (approximate) posterior. CTM is a different approach, which models that one topic can be correlated with another (LDA assumes they are independent). Instead of Dirichlet, it uses the logistic normal distribution, as I understand it. If you want to read the original paper for CTM, it is here: arxiv.org/pdf/0708.3601.pdf As far as spectral initialization, it is a good place to start and nice for getting quick and reasonable results. If I need something very robust, then I do all the work that is laid out in the stm package vignette. I am working on some tidy tooling around that, and hope to get it out sometime soon!

Thanks a lot for the quick reply and very useful info!

You can reduce the playback speed

Thank you

Everything is fine, thx. After restarting my computer my code is running ^^ Julia, u r great ^^u inspired me to do TM ..

Thanks for the response...I double checked and it is not a problem related to white spaces. I coded to get a igraph of bigrams and i get bigrams which are alone in two-nodes associations. Instead a bigram, there is a number on the empty node...

Yes, check out the topic modeling section of the workshop I taught at rstudio::conf this year: github.com/rstudio-conf-2020/text-mining

@@JuliaSilge Amazing! Thank you very much!

The first thing you need to do is read the Word files into R, because Word files are a special format that require specific handling. One package I like for dealing with Word and other Office files is officer: davidgohel.github.io/officer/ You can look at the same of the other options folks use here: stackoverflow.com/questions/50439684/how-to-extract-plain-text-from-docx-file-using-r

@@JuliaSilge thank you so much for your prompt response and for the resources you provided 🙏 I will definitely try them

For PDFs, my favorite tool for reading text into R is the pdftools package: docs.ropensci.org/pdftools/ I have less experience reading in .docx files, but I have occasionally used the textreadr package: github.com/trinker/textreadr Good luck!

Ol

It's been a while since I looked at this, but I don't believe there are any NA rows, at least as of how the data was formatted when I originally created this video/post back in 2018. You can see that in the tf-idf plot, no NA story facet: juliasilge.com/blog/sherlock-holmes-stm/

@@JuliaSilge Thanks for replying. Don't we select only the top 10 words on each document to plot td-idf? Oh! eventually NA is not that frequent. You are right, we may no need to remove NAs. Thanks again for the amazing material.

Yep! This example shows using stm for topic modeling with long documents (books) but this approach also works with shorter documents. If you want to see an example of this, I have a blog here implementing topic modeling with Hacker News posts: juliasilge.com/blog/evaluating-stm/

@Julia Silge yes, I've already checked that paper but I don't find specific information that related to the formula I mention, does the algorithm on estimating theta and beta matrix is similar to any topic modeling methods (i.e LDA, CTM, STM, etc)? thanks for the previous reply btw :)

If you look at the gamma probabilities, you can see how the stories are related to topics. Check out the plot "Distribution of document probabilities for each topic" here: juliasilge.com/blog/sherlock-holmes-stm/

@@JuliaSilge thanks, I'm gonna check that out...

I think you're dealing with some typos there; there's just one "t" and no "e" at the end: cran.r-project.org/package=gutenbergr

Yes. My mistake. Apologies. Thanks for correcting it.

HAHA I can't tell if this is serious or not 🙈 In case it is, I will say that since topic modeling is unsupervised ML, it can't be used in a straightforward way to evaluate better/worse (you are not predicting a label). Instead, like you say, you could compare the relative proportion of certain topics (like, say, a topic that seems to be mostly about topic modeling) in one video's comments compared to others, and make an evaluation of videos based on that.

​@@JuliaSilge If I can "cluster" comments related to topic modeling and find that the most relevant ones are linked to your video ID (based on beta, which will give you the top word probabilities), your video will appear with the highest relevance to that topic (based on gamma). This means your video is the most representative of that specific topic. But wait.. Then, if I manually compare, say, the top 10 most relevant videos and see that your video (which is at the top) also has a lot of likes, comments, engagement, and perhaps a great sentiment (after computing it) compared to the other 9, I can conclude that your video is the "best" and would recommend it. Does this make sense, or am I misinterpreting the gamma/beta. ***Assume I have concatenated all comments into 1 corpora. Each corpora is linked to a video ID.

@@dinohadjiyannis3225 I think that makes sense! Sounds to me like you are interpreting correctly. 👍

@@JuliaSilge A big thanks to you for replying, given that this video is 6 years old. 🥇