Sign language detection with Python and Scikit Learn | Landmark detection | Computer vision tutorial 

Thank you! So glad you enjoy the content! 😃🙌

Could you tell me the installation process

Good organization is the key to a successful project I am happy you enjoyed the video! 😄🙌

Thank you for your support! 😃🙌💪

Thank you for your support! Glad you enjoyed it! 😃🙌

can you help me. when I trained only one gesture nothing else, but the system detects untrained gestures as the trained gesture why? any idea

can you please share the changed code

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

Glad the content is helpful! 😃🙌

Thank you! Glad you enjoyed it! 😃💪

You are welcome! Glad it was helpful! 😃

Please send your github link please

I got lot of error bro please please please please

bro can you send me the file for your project

@@ijaspr5486 like the whole file?

Could you share your GitHub link of your project?

@@e2mnaturals442 yes like github code or i give you my social media id

You are welcome!! 😃

You are welcome! 😃 Glad you enjoyed it!! 🙂🙌

Glad to hear it is helpful! 😃🙌

size.width>0 && size.height>0 in function 'cv::imshow' error sir

I will try to. 🙌

Thank you, Arif! I am happy you enjoyed it. 😃🙌

Do you mean using Yolo for object detection instead of mediapipe + Scikit learn? It can be done. You just need to train it. I did it with mediapipe + Scikit learn only for simplicity, and I think it also results in a more robust classifier. 🙌

Hey Hayat, I am glad you found it helpful! 😄 Do you mean making a video about how to be strategic when starting a project and choose the most promising approach? Sure, I can do a video about problem solving strategies! 😃🙌

Do you have file word report ?

No particular reason why I used a Random Forest, I think pretty much any other classifier would have a similar performance in this case.

@@ComputerVisionEngineer Thanks felipe!!

am also thinking the same. The images seem no to be there

Hey George! Yeah, I sent that change in a new commit. It makes the solution more robust, you could think about it as a way of 'normalization'. This makes the classifier learn better than the (x, y) position of each landmark is not that important, the distance of each landmark to each other landmark is what matters most! 😃💪

@@ComputerVisionEngineer Thanks a lot for the answer! I thought it has something to do with the mediapipe library and is a must, but it actually makes sense to be some kind of normalization. Thanks for you time!

🤞😀 Good luck with your project, Martin! 🙌

@@ComputerVisionEngineer Sir i have an error "ValueError: The least populated class in y has only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2. ", what can be the problem, im trying to classfy all the alphabet letters, your help will be highly appreciated.

Hey, you can create the model yourself following the steps I describe in the video. 😃🙌

You would need to make some edits in order to use it with both hands but I guess it would work, yes. Regarding the performance, yeah you could train it and improve it so it can be used at a production level. 🙌

thanks @@ComputerVisionEngineer 😁i'll try and see if it works out

Hey @@ComputerVisionEngineer , its not working efficiently for the british sign lang , maybe because it uses both hands , do you have any suggestions on how i can build up my project , it'll be a huge help , thanks

The thing about Apple is that MacOS often puts a file called ".DS_Store" in the directory which stores some information. In your code where you iterate over folders, compare the name with ".DS_Store" and simply skip it

You may try to contact me in our discord.

Try using the method 'predict_proba' instead of 'predict'. You wil get a probability vector for all the classes. Taking the largest number will give you the confidence value you are looking for. 💪💪

@@ComputerVisionEngineer Thanks a lot you are amazing !!! 😃

@@prathamupadhyay1265 bhai if u dont mind kya app apke code ki zip file mujhe share kar skte hai, coz im getting many errors and i have tried many steps but kuch ho nahi raha hai. PLZ!!!!!!

plz bhai

@@yashanchule9641 GitHub link is there..or have you tried that too?!

Yes, you could try with video classification. 🙌

did you fix this error?

Hey, limitation in terms of possible symbols? I would say any static symbol made with only one hand.

If you're getting this, that means you're showing something else that isn't in the data. Only show what you've captured. Or else simply increase number of classes and take different pictures from different angles.

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

Thanks a lot!! How did you notice that this was the issue?

thanks a lot bro!!!

Hey, thank you! I am using mediapipe as a hand detector and landmark detector and a Random Forest classifier as sign classifier. 🙌

Hey, I didn't use any research paper for this project. 🙌

@@ComputerVisionEngineer alright then...but do you have any idea which one would be similar or near to this?

@@054_vishwadhimar4 hi, did you get the research paper?

@@aakritityagi7203 no I did not actually... thankfully my mentor did not force me to find one and accepted multiple youtube.videos as references

Hey, yeah I think that would work for a video for example, where you have already computed the object detection and classification for that video beforehand. If you are detecting objects live, in a webcam, you would need an object detector and an image classifier. I don't think you could load the annotations from a txt file in yolo fomat because you are capturing the data live. 💪💪

@@ComputerVisionEngineer Thank you for your response! I may have made my last comment a bit confusing. Basically, I am trying to adapt this code to eliminate the media pipe elements. I already have a dataset of still images and corresponding text files holding the bounding box coordinates. I am using scikit to train still. But I keep getting errors when i run the training script. Your help would be much appreciated

@daisybristow7036 oh I see, yes sure you could use something like yolov8 and replace both mediapipe and Scikit learn. Once you collected the data, you can just train yolov8 following the steps in my video on how to train yolov8 on a custom dataset. Then, for inference, take a look at my video on object detection + tracking. 🙌

@@ComputerVisionEngineer This would be an optimal solution, however I am required to use scikit to train not yolo. Would it be ok if I could explain it further beyond the comment section of this video? 😵‍💫

@@daisybristow7036 you could use both Scikit learn and yolov8, so you use Scikit learn to validate the class you got with yolov8, that would be a super robust solution! 😃 I may create a discord or something similar later on, for now it is only the comments section of my videos.

Hey Sourabh, it sounds like a complex and very cool project! I would start by saving all the symbols you detect, its confidence score, and the duration of time you detect them so you can analyze this info later on. This is going to help you to understand the problem a little better and also it is going to help you to define rules in order to achieve your goal. 😃💪

Hi Sourabh were you able to make it if yes could you please share some update or code for the same

Sure. Do you think you can take an approach as I do in the video?

Yes, by the way we are doing it in this tutorial, it should work if you change the background. 🙌

@@ComputerVisionEngineer i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

You are welcome! I will try to make a video about it. 🙌

hello I am having the same problem. Could be share your solution if you find one. Thank you!

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

😃 That is soooo cool! I am happy to help you guys. 😊🙌

Have you manage to Train the model with all alphabet letters

Hey, what file / line triggers this error?

Hey, how many different symbols are you trying to classify? How did you collect the data for each symbol?

@@ComputerVisionEngineer I change number_of_classes to 5 and i collect data through opencv by capturing images(by using the method describe in this video) Note: python version 3.11.2

total 5 symbols each got 0 to 99 images

There is a probably a bug with the data. Take a look at 'labels', how many elements are there for the different classes? Is it an array of integers or is it other data type?

@@ComputerVisionEngineer Now i am getting this error when i make 25 classes(for each alphabet). data = np.asarray(data_dict['data']) ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2471,) + inhomogeneous part.

Once the model is trained, anyone can sign the alphabet and it will show what sign they are doing.

Did you solve this error??

@@lal6797 if you have solution please inform me

@michaenrangelgiraldo5428 Okay so, I'm assuming that you are getting this error when predicting for that I just put an if condition like: if (len(data_aux) != 84) And with in that if condition I predict the values. I myself don't know whats causing this error but my assumption is it has something to do with the both left and right hand landmarks (42+42=84). Nevertheless, this solves this issue hope it will help you too.

Can you try this code: desired_length = 4200 # Pad data_aux with zeros to achieve the desired length while len(data_aux) < desired_length: data_aux.extend([0.0, 0.0]) # Truncate data_aux if it exceeds the desired length data_aux = data_aux[:desired_length]

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

Hey, not sure what could be going on, although it is always a good practice to take projects one step at the time. Try to do it with only 2 or 3 symbols and work your way up. It will make things easier to debug. 😃🙌

@@ComputerVisionEngineer Thank you. I took your advice and was able to fix the problem by breaking it down. Turns out the data for 3 letters were not properly captured but I re captured them and the 26 letters are working perfectly!! Thank you.

@@user-mh6ek3hv3k Amazing! Happy to hear you solved the problem! 😃

@@user-mh6ek3hv3k i am actually also facing the same error how can i identify which letter data is not captured correctly? pls can you tell me

@@foru1854 What I did was start with the first letter (A) , did carried out all the steps and trained the model. When I saw it worked, I added the second letter and did the two; then the third and that gave me the error so I knew the third had a problem and recaptured. I followed on like that and when I add a new one and I get the error I will know that alphabet needs to be recaptured. Hope that helps

I am looking at the Indian sign language alphabet and I see some characters are done with 2 hands and others with 1 hand. In order to do something based on landmarks as we did on this video you would have to train 2 classifiers, one of them taking as input the landmarks of one hand only (as we did on the video) and the other classifier taking as input the landmarks of both hands. Then some logic to apply one classifier or the other one depending on how many hands appear on the frame. Or, you can just follow a different approach and train an image classifier taking the crop of the hand/s. 💪🙌

Hi ! Have you completed working on this project? Did it worked ?

Do you mean the accuracy of the hand detection?

@@ComputerVisionEngineer yes

@@ocelottes it is mediapipe hand detection, if you want to test it's accuracy you would need to take another hand detector to compare mediapipe detections against

hi were you able to solve this? i used import matplotlib.pyplot as plt import seaborn as sns # class names class_names = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'] # Plot the confusion matrix plt.figure(figsize=(15, 15)) sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=class_names, yticklabels=class_names) plt.title('Confusion Matrix') plt.xlabel('Predicted') plt.ylabel('True') plt.show()

Sir kindly help me with this error. I am working on this project as my final year project and I have to extend it as my major project work.

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

Python 3.7 if not mistaken

Hey, have you installed the project requirements?

@@ComputerVisionEngineer yes, even the specific versions

i got the same error, were you able to solve it?

Did u solve it?

i was able to sort it using padding if you want me to explain more, i will be glad to

@@e2mnaturals442 please explain

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

did you figure it out?

Hey, yeah sure the project can be adapted for 2 hands. 🙌

@@ComputerVisionEngineer okay sir thank u🙏

same here, did you fix it?

If you trained the model using 2 hands, the issue may be that the in inference the model capture only one hand in a frame and that triggers the error. You could add an if statement to only run the inference if 2 hands were detected.

Can you please write the modified code?

i used padding to sort this out

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

hey did it work then?

​@@fruitpnchsmuraiG Yes, the program starts, detects the gesture, but after a minute it shuts down and gives the above error

i think guys to solve this problem we had to tell the collect data script to save just frames where he could detect our hands else we will store bad models that will ends with this getting errors like "inhomogeneous shapes" , i actually try to solved this problem by not moving my hand when collecting data and making my model else you can try this code to check your images that are stored This script will only print the paths of the images that are deleted due to no hands being detected. It won't display any image windows. ##########################################" import os import cv2 import mediapipe as mp def process_and_show(image_path, mp_drawing): mp_hands = mp.solutions.hands hands = mp_hands.Hands() # Read the image image = cv2.imread(image_path) image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Detect hands and landmarks results = hands.process(image_rgb) if not results.multi_hand_landmarks: print(f"Deleted image: {image_path}") # Delete the image with no hands detected os.remove(image_path) # Path to your data folder containing subfolders data_folder = "data" mp_drawing = mp.solutions.drawing_utils mp_drawing_styles = mp.solutions.drawing_styles # Iterate through subfolders for folder_name in os.listdir(data_folder): folder_path = os.path.join(data_folder, folder_name) if os.path.isdir(folder_path): print(f"Checking images in folder: {folder_name}") # Iterate through images in the folder for filename in os.listdir(folder_path): if filename.endswith(".jpg") or filename.endswith(".png"): image_path = os.path.join(folder_path, filename) process_and_show(image_path, mp_drawing)

Thank you! I haven't tried to do it an edge device, I don't have any suggestions. 🙌