How to use BERTopic - Machine Learning Assisted Topic Modeling in Python 

Thanks so much!! =)

Haha! That is so perfect! Hope this video helps!!

@@python-programming Definitely helped!

Thanks! Indeed you can. BerTopic has changed a bit since I made this video, so I will have to check the docs but I am certain you can.

Thanks so much! I am planning a new video on BERTopic soon to cover its new features.

I am so glad it was helpful!

Thanks! And great question. If you are looking for an off the shelf solution try top2vec, but I think you may run into similar issues. What language are your docs? Also, how varied are they in size? A more custom solution may be necessary.

@@python-programming I've standard English web sites, from product reviews to travel reports. Generally a page contains some 10 paragraphs. Content on a page is highly correlated (you'd expect), so the page content is defined by a few paragraphs. The topic of a paragraph is mostly in a single sentence; the rest is "glue". This turns out to be a reasonable assumption (eye balling). BERTopic supports these observations, especially if you remove paragraphs with the topic probability for the most dominant topic less than some cutoff (say 0.6; the reason that, worst case another topic is present for at most 0.4). From experience you're left with 3% unallocated documents; each allocated document has at most 3 topics. This is all nice, assuming BERTopic gives good results for both long and short paragraphs with the same hyper parameters. If my assumption is incorrect I've a problem :( So, thoughts?

Thanks for the question! You absolutely can. I have a whole other tutorial that walks through each of those steps. I think BERTopic, LeetTopic (my library), and Top2Vec provide a simpler solution for those who may not be familiar with a custom UMAP, HDBScan workflow. I try to make tutorials for users at all levels and I think these other libraries address the needs of those newer to Python/ML.

Yea that is a bit normal woth BERTopic. I plan to do another video that compares dofferent topic modeling approaches and that will be a key feature

@@python-programming Great, I’m looking forward to that video! 😊

Thanks! I would not use BERTopic, rather soaCy for text classification. You could use BERTopic to gather data for easy labeling.

@@python-programming thanks a lot, I actually went on to search for it and found another one of your videos explaining EXACTLY what I wanted! For reference it's this one: "The EASIEST! way to do Text Classification with spaCy and Classy Classification" thanks again!

@@luiztauffer8513 haha! Perfect! No problem!

BERTopic or Top2Vec will both work, but you'll need to reduce your corpus to shorter text. You can use an introduction or conclusion as your text, or perform some summarization before you start modelling

As long as there is a BERT model for Arabic, yes. I know there is an NEH funded project for this but I am not sure if it is available yet. There is a lot of research in Arabic NLP so I would be surprised if another does not already exost. I do not have Arabic, though, so I cannot validate the results.

@@python-programming Thank you for answering.

Thank you so much for your support and this wonderful comment!