How to Fine-tune LayoutLMv3: Fine-tune LayoutLMv3 with Your Custom Data | Part -3 Fine tuning 

Hi, Thank you very much for the support, Happy to see this comment. I have uploaded the inferencing part 4 as well, Please check out let me if you have any errors, queries you can comment in the respective videos, I will try my best to resolve it. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-MnpJKKSYJDw.html Thanks you again.

@@AIOdysseyhub error i m getting Traceback (most recent call last): File "d:\Python\visual-ocr-label\src\main.py", line 35, in train_loss = train_fn(dataload, model, optimizer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "d:\Python\visual-ocr-label\src\engine.py", line 11, in train_fn _, loss = model(**data) ^^^^^^^^^^^^^ File "C:\Users\Saujanya Basnet\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch n\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Saujanya Basnet\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch n\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "d:\Python\visual-ocr-label\src\trainer.py", line 33, in forward loss = loss_fn(output,lables) ^^^^^^^^^^^^^^^^^^^^^^ File "d:\Python\visual-ocr-label\src\trainer.py", line 12, in loss_fn return nn.CrossEntropyLoss()(pred.view(-1,4),target.view(-1)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Saujanya Basnet\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch n\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Saujanya Basnet\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch n\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Saujanya Basnet\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch n\modules\loss.py", line 1179, in forward return F.cross_entropy(input, target, weight=self.weight, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Saujanya Basnet\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch n\functional.py", line 3053, in cross_entropy return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: Expected input batch_size (1664) to match target batch_size (512).

If your code works can you please help??

Thanks for the support, Sure will share the inferencing code. Please subscribe to the channel more such related videos. Thanks again 😃😃

@@AIOdysseyhub yes. I did already. In the mean time can you please point out to some code where we can explore in the mean time

You can do it, but ocr will make a single bounding box for the multiple words but you can label each of the bounding box like each word as specific label that you want to do it for multiple words I hope you got my point, if not please let me know Thank you for the support and for reaching out me, please subscribe the channel 😄🙏

Thanks for the kind words 😌, no need to multiply trace back the error. Recheck it again cause I have done scale up and scale down for the box. Once check if you are giving the variable name correctly. Thanks for reaching out over comments. If you still stuck with the same error. Please let me know. Will try my best to resolve this. Thanks again, please subscribe to the channel for more such videos 😄👍

Sure, will release ASAP.

Please do it ASAP....really helpful@@AIOdysseyhub 🙏

Please choice the correct tokenize check if the variable name is correct or not.

I really appreciate your awesome support and the kind comment! It means a lot to me. 😊 I'm excited to let you know that I'm hard at work creating the video you've been looking forward to. Your patience is truly valued, and I can't wait to share it with you. Stay tuned for some great content coming your way! Thanks again for being such an awesome part of our community.

Great tutorial! Looking forward to learning how to load the model and how to predict classes on new document, in your next video!

Yes. Please share the inference part as well. soon.

Please convert your labels from strings to integers( one hot encoding) as mentioned in video, please let me know if it's doesn't help you

Thanks...yes .yday I noticed it integer was in double quote. After removing double quote it worked.

@@AIOdysseyhub Hi Mani i was also facing the same error "ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected)." I have also tried this solution but its not working for me . i already did the one hot encoding as mentioned in the video plz help with this.

while loading the model you need to use torch.device('cuda') to train with GPU

Sure, i will explain how to do inferencing with the trained model, I am also working on Drag GANs so it's taking time, sure will upload the video and code as well. Thank you very much for the support 🙏. Please feel free to provide feedback. Like the videos if you like it. 😄

You can't put the whole main in a cell, because there is no main the scripts need to work into a succession, you have to break it down into functions and isolate input and output of files in the temporary folder or in you google drive that you need to mount and run everything sequentially. You are better off using his code, it needs a couple of tweaks but it's all together working. In the file that produces the label studio json for labelling there is twice a function for creating a url, url that you need to get right because it will be used by label studio to render your image.

Is this resolved??

Hi @yeojinkim5100, In the src code I have not added evaluation metrics and the actual evaluation metrics used in LayoutLMv3 paper was map iou 0.50:0.95, you can use F1 score as well you can customize the code as per you wish, In the video I have just mentioned how we can start training process only, I have actual annotated more around 500+ images in the org and as we cannot share that data but the result were amazing you can also implement it using code mentioned in the video. Thank you for the support and If you have anymore questions please feel free to ask. Thanks again 😊😊

@@AIOdysseyhub will this work key value relationship extraction ? Bcz in Funsd Data for every form in annotation file has something called as "linking". It /establishes relation between two fields but i dont see that kind in your data set. Also you should have show the inference too after training the model bcz i don’t think this will work correctly for key values Extraction.

@@tejareddy4416 Hi Teja, Linking of key, value pair will be done in LayoutXLM model which is advance version of LayoutLMv3 model not in this LMv3 model. For showing inferencing I need to label more image for training, but in video I have just labelled 10 images just to show how we can start training using layoutLMv3 on our custom data. I have train LMv3 model with 500+ for my org data which is confidential. But it worked we were able to extract the exact information what we want but linking will not be happening, for that we have applied custom logics based on coordinates values Thanks for watching the video and comment. Please like and subscribe the channel in further video will improve the quality of content and upload more AI related videos Thank you again

Does it work only for English language??

@@moetezsoltani2182 No It will work irrespective of the language but while training and in inferencing we need to use OCR model which supports the language you want to extract. Thanks for the comment and please like and subscribe the channel for more such videos and in further upcoming videos will improve the quality and content of videos. Thanks for the support again.

yes

Hii, Thanks for the kind words ☺️ Have you converted labels in json file to integer like one hot encoding, I think that is reason for errors, if yes then check weather they are correctly mapped or not, then also you are getting same error let me know Thanks again for reaching out in comments 😊

Hi @AIOdysseyhub . Yes I did the one hot encoding part. I used 6 labels instead of 4 here and I am working on invoice dataset. Here i will provide the link of my 'Training_json' file : drive.google.com/file/d/1zbRoKxabZ8Uc3kr56c-a9IBSeyKIaPdF/view?usp=sharing

got any result?@@user-cq5wl1jk5r

Did you manage to find a solution for this? Im stuck here even after one hot encoding

​@@user-cq5wl1jk5r were you able to solve it? I'm getting the same error

Hi Abhi, Sure will search for you and if I find any will share it. Thanks for the comment and supports Let me know if we want to make video on donut as well

because model cannot understand direct text, so we are manually doing one hot encoding

Thank you for quick response. I realised it when running it with string.

Hi, We split into three sets (train, test, valid) if we want to test the model performance while training, by only splitting into two we are testing the model performance after training is done for that iteration. Thank you for the support. Please subscribe to the channel for more such videos.

HI Shreyan, Thank you for reaching us through comments, Sure will make the inference video as well, Please subscribe to the channel for more AI/ML related videos Thanks again.

@@AIOdysseyhub Yes Sure, I have trained the model but I don't know what's my out put is going to be, From last few days I am waiting for the upcoming video

Please check the json file you got from paddleocr output in that json file you need to do onehot encoding manually string to integer as explained in video. Let me know if it working or not. Thank for your support, Please subscribe the channel for more such videos.

@@AIOdysseyhub thanks for the response. i did exported to json-min format. changed the labels to integer manually. but still getting the error the only difference is that i have 5 classes and you demonstrated only 4 classes.

@@RishabhGupta93 number of classes didn't matter, it should be something with code, please post the complete error what you are getting

Sure, you can convert jsonl into json and then train it, I am not sure directly if we can use jsonl you can give it a try

@@AIOdysseyhub yeah sure I'm trying with jsonl I'll tell you after it works

Sure, I am on it

Hi, I have uploaded the fourth part inferencing, please check and let me know if you stuck somewhere I will try my best to resolve it Thank you

Please track back to which line the error was raising, based on that line we can check where we are doing mistake. If it's not helping, please let me know. Thank you

I have resolved this issue, it is in the input json file, instead of the Key Label being of type str I have not changed it to type int. Thank you very much! Can I contact you via social media?@@AIOdysseyhub

@@AIOdysseyhub facing same issue. Please help Some weights of LayoutLMv3ForTokenClassification were not initialized from the model checkpoint at C:/Users/AshwariyaSah/ASH/LayoutLMV3_Fine_Tuning/inputs/layoutlmv3Microsoft and are newly initialized: ['classifier.bias', 'classifier.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of LayoutLMv3ForTokenClassification were not initialized from the model checkpoint at ../inputs/layoutlmv3Microsoft and are newly initialized: ['classifier.bias', 'classifier.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. 0%| | 0/4 [00:00

@@AIOdysseyhub File "C:\Users\AshwariyaSah\ASH\LayoutLMV3_Fine_Tuning\src\main.py", line 35, in train_loss = train_fn(dataload, model, optimizer) File "C:\Users\AshwariyaSah\ASH\LayoutLMV3_Fine_Tuning\src\engine.py", line 9, in train_fn for data in tqdm(data_loader, total=len(data_loader)): File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\tqdm\std.py", line 1182, in __iter__ for obj in iterable: File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\torch\utils\data\dataloader.py", line 630, in __next__ data = self._next_data() File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\torch\utils\data\dataloader.py", line 674, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\torch\utils\data\_utils\fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\torch\utils\data\_utils\fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\AshwariyaSah\ASH\LayoutLMV3_Fine_Tuning\src\loader.py", line 32, in __getitem__ encoding = self.processor( File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\transformers\models\layoutlmv3\processing_layoutlmv3.py", line 122, in __call__ encoded_inputs = self.tokenizer( File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\transformers\models\layoutlmv3\tokenization_layoutlmv3_fast.py", line 330, in __call__ return self.batch_encode_plus( File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\transformers\models\layoutlmv3\tokenization_layoutlmv3_fast.py", line 412, in batch_encode_plus return self._batch_encode_plus( File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\transformers\models\layoutlmv3\tokenization_layoutlmv3_fast.py", line 670, in _batch_encode_plus return BatchEncoding(sanitized_tokens, sanitized_encodings, tensor_type=return_tensors) File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\transformers\tokenization_utils_base.py", line 223, in __init__ self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis) File "C:\Users\AshwariyaSah\.pyenv\pyenv-win\versions\3.10.0\lib\site-packages\transformers\tokenization_utils_base.py", line 764, in convert_to_tensors raise ValueError( ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

Hi @@user-rx7td3lr4i, Are you getting this error while inferencing, or while training the model ?

Sure, I will

Hi, Uploaded the inferencing part, Please check and let me know if you stuck some where ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-MnpJKKSYJDw.html thank you for the support

Hi, yeah I can understand that, I am making another video from layout lm paper explanation to coding in depth, will try to increase the quality at Max. Thank you for the feedback and please support the channel 🙏

@@AIOdysseyhub hey...when can we expect the newly recorded video where we can listen properly?

@@sankettgorey91 Hii, We will release with in two days, Thanks you awaiting.

@@AIOdysseyhub also please upload the video for inference. its important as trained model without deployment is of no use.

@@sankettgorey91 yeah sure will add that as well 👍