Sign Language Detection using ACTION RECOGNITION with Python | LSTM Deep Learning Model 

I remember some time ago requesting this type of video,but to see that its finally here brings me joy. Can't wait to do this and show to my sign language friends.

00:01 This video demonstrates sign language detection using action recognition with Python. 01:40 The video discusses the process of sign language detection using action recognition and LSTM deep learning model. 05:16 MediaPipe Holistic allows us to get key points from face, body, and hands 07:17 Setting up webcam access and rendering frames using OpenCV 11:06 The code captures frames from a webcam and displays them on the screen. 12:46 Setting up MediaPipe Holistics and creating variables for MediaPipe Holistic and MediaPipe Drawing Utilities 16:46 The video explains the process of color conversion in sign language detection. 18:32 The process involves detecting sign language using media pipe and a deep learning model. 21:59 The video discusses the different types of landmarks in sign language detection using action recognition. 23:23 The video explains how to detect and visualize different types of landmarks using MediaPipe. 27:05 The video discusses how landmarks in facial and body pose can be connected to each other. 28:37 Implementing sign language detection using LSTM deep learning model in Python 32:18 Landmarks are drawn and rendered in real time using image pass and cv2 33:55 You can customize the formatting of the dots and connections in Sign Language Detection using a landmark drawing spec and a connection drawing spec. 37:32 Updating pose and hand landmarks with different colors and parameters 39:31 Different models in action: left hand, right hand, face, and pose. 43:09 The code demonstrates how to extract landmark values using pose estimation. 45:04 The video explains how to reshape and convert landmarks into a single array. 48:27 Building a neural network and extracting key points using action recognition with Python 50:10 Setting up error handling and placeholder arrays for pose and face landmarks. 53:52 The video explains how to extract key points for sign language detection using LSTM deep learning model in Python. 55:31 Concatenating pose, face, left hand, and right hand keypoints for sign language detection. 59:11 Using LSTM Deep Learning Model to detect sign language actions 1:00:57 Creating folders to store data for different actions and sequences. 1:04:16 Creates a folder structure for sign language detection using action recognition with Python. 1:05:48 Collecting data using MediaPipe loop and capturing snapshots at each point in time. 1:10:14 The code is outputting text to the screen and taking a break at frame 0. 1:11:44 The first block of code prints starting collection in the middle of the screen and pauses. 1:15:10 The code collects key points by looping through actions, sequences, and frames. 1:16:39 Implementing sign language detection using action recognition with a LSTM deep learning model. 1:20:25 Sign language detection using action recognition with Python 1:23:20 Using MediaPipe to collect key points for sign language detection 1:26:55 Creating a dictionary to map labels to numeric ids 1:29:08 Sequences represent feature data and labels represent y data 1:32:26 Data preprocessing and training and testing partitioning are important steps in sign language detection using LSTM deep learning model. 1:34:12 Training LSTM neural network using TensorFlow and Keras. 1:38:14 The model uses LSTM layers for sign language detection. 1:39:55 The next three layers are dense layers using fully connected neurons. 1:43:16 The video discusses the process of formulating a neural network for sign language detection using action recognition and LSTM deep learning model. 1:44:58 Training the model with 2000 epochs 1:48:11 The training accuracy is high at 93.75% after 173 epochs. 1:49:37 The model has three LSTM layers and dense layers, with a small number of parameters to train. 1:53:13 Reloading a deleted model and evaluating its performance using scikit-learn. 1:55:03 Converting y test and y hat values to matrices and then evaluating the model performance using a confusion matrix and accuracy score. 1:58:18 Implementing prediction logic by concatenating data onto sequence and making detections when 30 frames of data are available. 2:00:30 Implement logic to grab the last 30 sets of key points for generating predictions. 2:04:18 Implementing visualization logic and checking result threshold and sentence length 2:06:45 The code checks if the current action matches the last sentence in the string. 2:10:05 Sign language detection using LSTM deep learning model 2:12:39 The video discusses sign language detection using action recognition with Python using an LSTM deep learning model. 2:17:13 The video discusses sign language detection using action recognition with Python 2:19:14 Sign Language Detection using Action Recognition with Python 2:22:29 To ensure accurate action detection, the last frame needs to be included in the sequence. 2:24:04 The code implementation adds stability by checking if the last 10 frames have the same prediction. Crafted by Merlin AI.

I can not thank you enough for all the videos you create i was a noob in tech but the moment i started watching your videos its been a year now and i am so proud of you and myself for coming this far and this project works for me❤

Thank you for these clear, practical, straight to the point tutorials! Looking forward to your future videos!

as someone who is following this in 2023, here's some change(s).....i'll be editing them in as they pop in while I go through the tutorial. 25:42 FACE_CONNECTIONS seems to be renamed/replaced by FACEMESH_TESSELATION.And well since we want just the outlines of the face, it's FACEMESH_CONTOURS that we would need in this project.

Man, you are so underrated and deserve a lot more! thanks a lot for these awesome learning materials! I have learned a lot from you. Keep inspiring, man :)

It's so cool to see how happy Nicholas is when everything works in the end. That's the spirit! Amazing video, thanks a lot for your work man!

Thanks for the amazing tutorials! absolutely life-saving. Just a reminder that the z value from mediapipe is with respect to the wrist landmark not the distance from the camera! I found out pretty late!

thanks for this amazing tutorial sir , we are working on a project that needed this section and your videos and explanation are being extremely helpful to me and my team ! thanks a lot

Thanks, great videos. Would be great if you could elaborate into the differences of the used media pipe implementation, compared to the others you mentioned. I mean really a comparison of the underlying models/ networks and their training.

Thank you @Nicholas Renotte I just passed my capstone project defense utilizing this deep learning model

Nicholas is the best machine learning youtuber, his tutorials are interesting and fun.

I was really waiting for this video!

Hi, Nicholas! These are great video series to watch and learn! Thank you very much! Can you please prepare a video applying CV on real-time sign language detection on the base of a ready dataset avaliable in Internet? It may be much more interesting if we can see ViT in action recognition as well.

Super stuff Nicholas! Super grateful for your tutorials 🙌🏻. Keep up the great work!

Great explanation on how to use LSTM with pose coordinates.

that's amazing! I watched this video more than a month ago but it seemed difficult for me as a beginner. Then I've tried my best to finished Machine Learning/ Deep Learning/ Python / Tensorflow and some Data Science course within a month. Now watching this video again is like watching a movie! it's easy to follow! love it

Hey Nicholas, I am working on similar project. Just wondering when I test the model using your metric it does not reflect the same accuracy as the real-time test. I train the model accuracy to 80-90% but the real-time test barely capture any sign language. Do you have any thought?

Hi ! I'm impressed by the amazing clarity of your explanations. For one second I thought you must be a trained teacher robot....

Hi Nicholas thanks so much !!!! I am creating a model to help deaf people here in my country. Greetings from Guatemala !!!

I have a question. Is it possible to feed video of arbitrary lengths (frames) instead of feeding an action of fixed length video? Because in real time, we perform sign language pretty fast and each action is of arbitrary length.

hey nick! this project is amazing! thank you for these awesome tutorials. You did sign language detection with tensorflow object detection which detects sign using single frame but here we're using multiple frames to detect it. So i was wondering how is this one better than tensorflow object detection?

This content is so underrated! Thank you so much!

Thank you so much Nicholas 😌 This will help me with my project 🙌🏾

Hi Nicholas, been working on similar project. I believe this tutorial is done for simplicity so I would like to add a piece of my mind. When adding more action, the prediction on the realtime will be mix a lot due to frame overlap and wrong slicing of the frame. I would suggest to show some viz as start and end of the prediction. So the user can follow the start to the end frame. This way it is similar to the way it is collected and higher prediction accuracy.

This is an amazing video! I have learned a lot following your tutorials. One question - What if you are trying to capture actions that are not of equal time duration. E.g if a sign language like "Good Morning" lasts for 5 seconds and another sign like "Welcome" lasts for 9 seconds. How can this be treated?

Exceptional tutorial. Thank you from the bottom of my heart.

Thank you for a detailed and wonderful mediapipe tutorial :)

This is so awesome! You can theoretically do the same for the static letters in the ASL alphabet, right? Just make the dataset that contains each hand sign. The model would be train on the keypoints of each hand sign. Although I am not sure it for some hand sign letters, the keypoints would be accurate. What do you think?

Hi Nicolas, Such an amazing video. Helped me a lot building a project. I am working on a different project in which I trained the model with around 20 signs from ASL but I am getting a categorical accuracy of only 0.05 on average in each epoch. can you tell me where I went wrong or anything to imporve it?

That's way cooler one and your explanation was a ton easier 😉 Thanks Nick

Videos that you make is wonderful. Thank you for these practical and clear points in the tutorials.

Greetings Hey man this is an awesome tutorial and I completely love the way u have explained the process step by step. It was an awesome tutorial and I completely loved it. I tried it on my own and I have encountered some problems it would be a great help if u could help me out with it. I have followed the same method that u have prescribed on the video these are the problems which came up. Even after getting overall categorical accuracy 95% and above accuracy on training datasets when I do the gesture recognition it is not recognising one gesture. And sometimes it shows the same gesture even though I am showing a different gesture. Sometimes even it is detecting 2 gestures even though I am not giving any gestures. I am always retraining the same data to get a higher accuracy before going to the gesture recognition part. I have also increased a layer in the LSTM model but the results are the same. Would greatly appreciate the help.

Hi Nicholas, Thank you for this amazing tutorial. If we use padding in this case (videos containing movements with different number of frames) how can we make predictions in real-time? In the tutorial you set a fixed length (30 frames) (sequence=sequence[:30] if len(sequence)==30), but in my case there are different frame sizes for each activity in real-time prediction.

Great Tutorial Nicholas. Thank you so much !

Thank you so much! You are my best teacher in my college life!!!!

Hi nick! Nice tutorial on this sign language recognition program. I had faced some problems of the categorical accuracy staying the same when im trying to add more signs to the model rather than 3 like the ones you used in the video, is there any way to solve this issue? Thanks!

Can’t lie.. I have learnt a lot from Nicholas

Hi, Nicholas! I learned a lot from this video! Thank you very much!

Wow! This is great Nick! 👏👏👏

Hii Nicholas This is really an amazing tutorial I really appreciate it. But I am having an error at fitting the model and it is of 'ValueError' which is "Failed to convert a NumPy array to Tensor". I tried many things but it is not going away so please give me a hand on this.

you are great.

thank you Nicholas i have learned a lot of things from this video ....that I was looking for the past few months..

Awaited Video Nicholas hope to complete it and implement on my side.

Hey Nicholas, the tensorflow version of 2.4.1 is showing an error. It says " Could not find a version that satisfies the requirement tensorflow==2.4.1" please help me out

Hi Nicholas, thanks again for this great tutorial! I am writing this because I would like to learn more about how did you chose your architecture. As you mention, almost all the state-of-the-art papers use a combination of CNN and LSTM. So, I have two questions 1. Would it be possible to get a more detailed explanation on how you build this model? 2. So far, I have reviewed “LSTM: A Search Space Odyssey” by Greff (+ other papers) and the “Neural Network design” handbook by Hagan. Could you recommend me any documents regarding LSTM architectures?

Helped a lot.. Everything explained really well.. Thank you so much!

Your tutorials are really very productive. Why don't you start your courses for tensorflow, python, etc . on some online learning platforms ?

Hi Nicholas! Thank you for your tutorial once again. Quick question, How can I change the code to access a folder that contains a dataset of videos without live capturing them? Really appreciate if you could give an answer :)

Great tutorial as usual, Nick! Learning a lot from you :D Did anyone try to use a set of videos instead of recording their own videos? If yes, what did you change in the code base? Kudos

It was really a great tutorial on real time sign language detection.

Amazing tutorial! Would love to see action recognition done with video transformers!!

Is there a dataset available for this, which has more signs than these 3. If so it will be really great if someone could share it

Hi Nicholas, thanks for the awesome tutorial! I've got 3 questions about the project, hope you don't mind helping me: 1. When training my model, i've 90%+ accuracy very quickly (150 epochs more or less), but all of sudden it dropped to 30% and kinda stabilized until the rest of the execution, how can I fix it? 2. If I want to add more signs after first training my model, I'll have to re-train it? Or can I train just those specific signs separately? How do I do it? 3. After the model is working just fine, it is possible to attach the real time script to an android app?

Your every video slightly edited , directly goes in my cv 🤣🤣🤣🤣 , thanks for taking care of my future ❤❤❤

Thank you for this this helped me so much with my project esp making the dataset

Hi Nicholas, I've been working on my thesis project about the quality of body movements and I encountered a problem with keras. I see that you feed in the first layer a sequence of constant 30 frames (1 second of video/list of mediapipe landmark object). In my case I have a variable number of frames (i.e. a video containing movements that lasts some 2 seconds (60 frames), some 2.5 seconds (75 frames), some 3 seconds (90 frames), etc., hence with different number of frames), how can I solve this? I looked around and people say that I can apply the so called "padding and masking" which takes the largest number of frames (longest video) and then add a special value to the others (padding) and after that somehow ignore/filter the special number later (masking). But this can't be applied to my case because I would like to have the freedom of variable number of frames during prediction. I hope you understand what I want to ask, otherwise let me know and I will try to clarify it as much as I can. Thank you!

لا يمكنك أن تتخيل كم ساعدتني. شكرا جزيلا لك 🙏🏼

Excellent instruction! TY very much for sharing! :)

Update: I managed to get an accuracy of 98% by changing the activation functions of the LSTM and Dense layers. 😌 Hope that this helps y'all who might be stuck on this Hi Nic 😌 me again 😅 So I'm trying to use a new data set of gestures and I can't seem to get an accuracy >20%. I have tried to change the learning rate, the optimiser, etc, but non of these work 🙈 Is there something that I am missing? Thank you in advance 🌸

hi .. first of all thanks for your free code to this brilliant work. second, i am a beginner in using python yet i have come too far in running your code. At step 11 i am facing an error that keeps appearing and i am exhausted right now bcz i have spend my whole day in finding a perfecct solution for it. it keeps showing TypeError: only size-1 arrays can be converted to Python scalars. after running plt. imshow coding line ..... plz plz need help...

31:06 Im making a project to my school and he's look like your project, and he's function is like your. But, im dont have knowledge sufficient to make this alone. So im watching your video to learn much and complete that, thanks for the video and sorry for my english haha. Send hello to Brazil 🇧🇷 😂

Thank you so much, Nick! Your video is fantastic and I have to say your method of teaching is top-notch mate. I am using you video for my project to interpret all the 26 letter signs in ASL. Could you please do me a favor of telling me how to train the model using an already availble dataset instead of creating it ourselves, like how it's done in the video? Your help will be much appreciated. Even the suggestion of one of your videos where you use a dataset will suffice.

finally it worked

I'm deaf help

Brilliant as expected

Nice work, thank you for the wonderful video!

Always Great Videos :). I have learned many Things from you. Thx

Hai nicholas it was such a wonderful implementation and thanks for sharing it with us

Wow!! such a helpful video. Thankyou very much

Hi Sir. Could you please explain the basis of the model? Especially, on the part where you used multiple LSTM and DENSE layer as well as the number of units.

This helped me sooo muchhhh! Thanks.

Awesome video, thanks a ton for making it, it really helps beginners learn hands-on! A question I had, is it possible to extract the keypoint values from just the hands? I found that the ML model was detecting words even when my hands weren't in frame (i.e. just moving my head or face).

Nicholas certainly deserves more views.

Thanks a lot for this content, I've been learning a lot, you are a god :)

Very good video. Thank you for your hard work :) New subs

Someone just completed his internship with the help of your code and also got a certificate from an IT company

Very good prjoect , very well implemented ,

Thanks for this concise and insightful tutorial! One question: Isnt some normalization or standartization of the features/landmarks necessary before training the model? If not, coulr you explain? Thanks!!

Best lecture 😍 sir will please make a full video for beginners

Hey Nicholas! Best wishes for the new year! Really like the content you make, keep it going!! I wanted to ask two questions. 1. Is it possible to get landmarks according OpenPose keypoints instead of BlazePose (the one that Mediapipe uses by default), in Mediapipe? 2. If say I have 100 different sentences, so 100 different action sequences, would it be wise to implement it like the way it's implemented in the video or would you take a different approach? Just wanted to get some clarity in concepts. Thanks!

Wow, Thank you. I like all your video. You are very intelligent.

Thanks For Considering My Comments And Did with LSTM Love You and Really Good

Wonderful video. Keep posting such great content videos

Hey Nicholas, great tutorial really enjoy this content. Have you considered doing a video for a preview with Gradio? Would be great help If you do an in-depth much like this one.

Amazing tutorial sir...

Hi, Nicholas! Thank you for making this amazing video. Can you please make a video that how we can deploy this model in flutter

Another biggeeeeeee on the heap!,amazing sir❤️🙏

Great video Nicholas! I downloaded the notebook but am having issues with the different versions of the different packages. Which versions of the packages did you use? Thanks!

I wish you recorded all your videos with zoom and font size of 22:30, it is really great 🙂

Great work, thanks.

thanks for your presentation .

Hello Nicholas, I really enjoyed this tutorial. I wanted to ask if there was a way to normalize the x, y and z coordinates to they are not dependent on their position in the frame.

I really hope you continue this project.

Hi, Nicholas Thanks for this stellar tutorial. Is it possible to train multiple models at the same time?

Hi Nicholas, first I have to thank you so much for ur great awesome tutorials, I'd like to ask how can I reduce the fluctuation of predicting the action, even though ur case was almost close to zero, but in my case I got huge function, and sometimes even if I don't pose or do any trained action it will still predict the wrong pose

This is such a cool video! Thank you so much I do have two questions: 1. Are the keypoints normalize? I mean, do they have values relative to image size or are they regular values? 2. If I were to use static frames instead of sequences of frames, what type of layers could be used? I just want to detect static hand signs, whihch wouldn't need a sequence of frames

Hi Nicholas! Your video is amazing, it helped me a lot. I have the following question: Can we use this method to identify human emotions through facial points? If yes, what points should be paid attention to? Thanks very much.

Awesome tutorial as always Nicholas ! I am a huge fan! , please tell what changes I should do if I am doing this with just one class (binary classification) ?

Nicholas Renotte! Thank you so much.