Florence-2: Fine-tune Microsoft’s Multimodal Model

Подписаться 36 тыс.

Просмотров 7 тыс.

50% 1

Learn how to fine-tune Microsoft's Florence-2, a powerful open-source Vision Language Model, for custom object detection tasks. This in-depth tutorial guides you through setting up your environment in Google Colab, preparing datasets, and optimizing the model using LoRA.
Chapters:
- 00:00 Introduction: Unlock the Power of Florence-2
- 01:09 Getting Started: Prepare for VLM Fine-Tuning
- 03:55 Florence-2 in Action: Explore Pre-trained Capabilities
- 07:00 Dataset Deep Dive: PyTorch Data Loading for Florence-2
- 13:02 LoRA: Optimize Your VLM Training
- 14:21 Fine-Tuning: Unleash Florence-2's Custom Object Detection
- 17:30 Model Evaluation: Measure Your VLM's Success
- 21:37 Florence-2 vs Other Computer Vision Models
- 24:09 Conclusion and Next Steps
Resources:
- Roboflow: roboflow.com
- 🔴 Community Session July 3th, 2024 at 08:00 AM PST / 11:00 AM EST / 05:00 PM CET: roboflow.stream
- ⭐ Notebooks GitHub: github.com/roboflow/notebooks
- 📓 Florence notebook: colab.research.google.com/git...
- 🗞 Florence-2 arXiv paper: arxiv.org/abs/2311.06242
- 🗞 Florence-2 overview blog post: blog.roboflow.com/florence-2
- 🗞 Florence-2 fine-tuning blog post: blog.roboflow.com/fine-tune-f...
- 🔗 Florence-2 HF Space: huggingface.co/spaces/gokaygo...
- 🗞 Mean Average Precision (mAP) blog post: blog.roboflow.com/mean-averag...
- 🗞 Confusion Matrix blog post: blog.roboflow.com/what-is-a-c...
Stay updated with the projects I'm working on at github.com/roboflow and github.com/SkalskiP! ⭐

Наука

Опубликовано:

10 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 44

@abdshomad 9 дней назад

I've been waiting for this tutorial for days. Thank you again for being the first to comprehensively review this new model. Super exited! 🎉🥳

@Roboflow 9 дней назад

As usual you are the first one to comment on the video! Thanks a lot for all the support! 🔥

@jk_c66 7 дней назад

thank you roboflow for providing such nice and lovely tutorials for free and with a nice instructions

@Roboflow 7 дней назад

Pleasure!

@artem-yw8km 7 дней назад

Thank you for this turtorial, was working on these kind of setup for a couple of days. You definetely could save lot of time

@Roboflow 7 дней назад

Sad I didn’t save your time this time.

@SatyamKumar-cb2mt 8 дней назад

Thanks a ton for this awesome video! Every single term is explained so clearly-it's super helpful. I can't wait to dive in the code and start putting this knowledge to use!

@Roboflow 8 дней назад

Thanks a lot! I really put an effort and try not to fall into a bias (not assume that people know those things).

@SridharanS-vz7re 8 дней назад

how to train this model on custom dataset for OCR

@kylewang6704 4 дня назад

Thank you for the awesome tutorial! I wonder what about the detection accuracy comparing to YOLO based model?

@Roboflow 4 дня назад

We cover that topic in the video ;) something tells me you didn’t watch till the end.

@nikilragav 2 часа назад

9:35 how did you see this embedding vector projection thing for the Roboflow 100 datasets?

@kylewang6704 День назад

Do you have plan to release tutorial about finetuning Florence-2 on other vision tasks such as captioning? I wonder if Florence-2 can be tuned in a way so it can do object detection/classification and also provide the reason for its prediction.

@Roboflow 20 часов назад

That’s quite possible. We will release a KeyPoint detection video this week and if no new model come out we will drop one more Florence-2 tutorial.

@VLM234 9 дней назад

Very informative video. Thanks for making auch a valuable video free of cost. Just one request when your you make tutorials if possible try to do inferencing, training or fine tuning on agricultural or satellite related data.

@Roboflow 9 дней назад

Next time I will try to find some cool datasets from this domains

@user-mc7tg4pf3i 7 дней назад

Hell Sir Thanks for your all videos and efforts. I am following your channel, but I request you please upload one detail video on how to finetuning Yolov5 model for custome images classification.

@Roboflow 7 дней назад

Does YOLOv5 support classification?

@user-mc7tg4pf3i 4 дня назад

@@Roboflow Yes

@dabaizhang-x5b 7 дней назад

Master, could you please tell me if Florence-2 can perform SER (Semantic Entity Recognition) and RE (Relation Extraction) tasks? If so, what should my dataset look like? 🤔

@Jordufi 9 дней назад

Nice video, as usual

@Roboflow 9 дней назад

Thanks a lot!

@suphotnarapong355 8 дней назад

Thank you

@sandrojunioraraujo3706 6 дней назад

Wonderful tutorial! Could you make a tutorial about how to fine tune florence 2 for the segmentation task?

@Roboflow 2 дня назад

I'm almost sure I'll create Google Colab covering this topic. Not sure about RU-vid video.

@8-P 8 дней назад

For the community session I have a couple of (beginner) questions: - the google collabs on roboflow seem to be linux based, is there an easy way to make them work on windows? - in general, how do I download a model (YOLO) to use in a python app (on windows) - are there models that would run for realtime video detection on a regular laptop with an integrated iGPU? - I am planning to use a YOLO model for a sports live stream, but only have a simple 3 Year old mid range laptop on me - would it be better to send the stream over to my desktop PC with an Rtx3060Ti-8GB and let the model run there (and send back the detection back and sync on the laptop) - if a laptop is underpowered? - for simple applications, like the realtime sports detection of yours, would it be better to run it on my own hardware or investigate in cloud servers for inference? Thank you very much for your tutorials, the help a lot!

@3DFinalCut1 6 дней назад

Thank you for this very informative video. Something like this helps enormously. Dziękuję! I have a question and maybe someone here can give me a tip. I am looking for a tool that searches for similar images in a folder and shows me the results so that I can clean up the data set later. I have already found tools that do this, but they mainly work with image hashing methods or use fuzzy matching algorithms. But I wonder if there aren't already tools that use AI to solve this task. Does anyone use such a tool?

@Roboflow 6 дней назад

You can make it happen using CLIP model. We covered that topic here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-YxJkE6FvGF4.html

@UhuruSsemakula 6 дней назад

Can you please make one for Object detection using web camera?

@Roboflow 6 дней назад

You mean using Florence-2 and Webcam? Or webcam in general?

@UhuruSsemakula 6 дней назад

Yes Florence-2 and Webcam

@jimshtepa5423 7 дней назад

can you please upload recording of teh community session for those of us who are in different zone or might otherwise miss the call?

@Roboflow 7 дней назад

Sure! All our community sessions are available to re-watch on YT channel

@geniusxbyofejiroagbaduta8665 8 дней назад

Thanks Sir. Please do fine-tuning for Oct, captioning and segmentation task

@Roboflow 8 дней назад

Did you tried to run OCR with pre-trained model?

@yj1548 7 дней назад

Good video.and I'm curious about what can be done to improve mispelled class names on object detection tasks，do you have any ideas?

@Roboflow 7 дней назад

I think you asked me this question on Twitter, but let me answers here as well. 1. Longer training could fix it. 2. Fuzzy class matching. In the video we filter out anything that is not exact match. Hamming distance for example.

@bladethirst1 8 дней назад

Is this applicable to grade handwritten pdf math assignments?

@Roboflow 8 дней назад

Florence-2 can be really good at OCR processing of handwritten text. Not sure about math equations. We would need to confirm that.

@bladethirst1 8 дней назад

@@Roboflow {'': 'In this image we can see a book with some text on it.'} This is the test output of a handwritten math problem deduction, is there someway to get more detailed caption or the OCR output?

@geniusxbyofejiroagbaduta8665 8 дней назад

Please sir also tech us how to annotate with it

@Roboflow 8 дней назад

You mean how to automatically annotate images?