Тёмный

Face, Hand, Pose Tracking & More in TouchDesigner with @MediaPipe GPU Plugin 

Torin Blankensmith
Подписаться 8 тыс.
Просмотров 31 тыс.
50% 1

Hey! In this tutorial, we'll go over the new free open-sourced MediaPipe plugin for TouchDesigner that's GPU accelerated and works on Mac and PC with no instillation. The plugin supports face, hand, pose and object tracking with multi-person face detection, object detection, and image segmentation / background removal.
Link to the GitHub to download the Plugin: github.com/torinmb/mediapipe-...
Dom Scott's website: art.domscott.ca/home
Link to Torin's Patreon: / tblankensmith
Link to the body tracking alternative: • Tutorial 1/3 - New Bod...
Link to OBS for using video with MediaPipe on Mac: obsproject.com/
Link to SpoutCam for using video on PC: github.com/leadedge/SpoutCam/...
0:00 Intro and Overview
1:44 Project Setup
5:43 Face Landmark Detection
9:37 Hand Tracking + Gesture Recognition
13:49 Pose Tracking
16:37 Object Tracking
18:05 Multi Person Face Detection
19:21 Image Classification
20:56 Image Segmentation / Background Removal
23:58 On Mac Pipe Video into MediaPipe with OBS
26:26 On PC Pipe Video into MediaPipe with SpoutCam
28:30 Outro

Опубликовано:

 

1 июн 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 97   
@gianlucacolia3431
@gianlucacolia3431 8 месяцев назад
This plugin is amazing, I hope you keep updating it. thank you so much for sharing it!!!
@eldelbaar
@eldelbaar 7 месяцев назад
Thanks a lot for making this project public! it helps a lot for people that is starting
@pagarreton
@pagarreton 8 месяцев назад
Wow this is incredible! I am really happy it also works for Mac computers!
@TrevorNew
@TrevorNew 8 месяцев назад
This is amazing, thank you both for making this!
@devicenotfound
@devicenotfound 8 месяцев назад
Thanks for sharing such a helpful video! I never thought TouchDesigner could do this, and it's truly inspiring. Your video also has shown exactly what I've been looking for - how to superimpose TouchDesigner output to OBS.. Your effort is greatly appreciated!
@AliTanUcer
@AliTanUcer 8 месяцев назад
this is phenomenal!!... i was about the tackle the same thing and wishing to find time to port everything to GPU.. now we have it.. amazing.
@gregderivative2647
@gregderivative2647 8 месяцев назад
o my, brilliant tracking without special hardware. great explanations.
@jimkuehn
@jimkuehn 8 месяцев назад
Sick sick sick. Am rolling into a project that might benefit from your great work. Perfect timing
@benjavides
@benjavides 8 месяцев назад
This is incredible! Thanks for publishing it ❤
@markus_knoedel
@markus_knoedel 8 месяцев назад
WOW, I will be off grid now for some time to figure all of this out. Thank you. So helpful.
@user-hm2ne5zr6n
@user-hm2ne5zr6n 8 месяцев назад
Amazing, thanks for your hard work on this!
@BossCODINGplease
@BossCODINGplease 12 дней назад
This is truely amazing, thanks for all the hardwork
@EMERGENCIASTATE
@EMERGENCIASTATE 8 месяцев назад
you are the real master bro ! keep doing it !
@cele_stin
@cele_stin 5 месяцев назад
This is unbelievable, thank you guys!
@diyacorrea1023
@diyacorrea1023 8 месяцев назад
you guys are amazing thank you so much!!!
@carlosgaston3985
@carlosgaston3985 4 месяца назад
you two are so friking awsome! thanks a lot !
@saralopez7873
@saralopez7873 5 месяцев назад
This is incredible! Thank you so much
@feelzlab
@feelzlab 2 месяца назад
wow! I just get to know about MediaPipe - thank you so much for creating this and making this tools more accessible for individuals like me who have visual ideas but are not as technical as one can be.
@blankensmithing
@blankensmithing 2 месяца назад
Thanks! that manes a lot. Excited to see what you create with it!
@christopheluez1860
@christopheluez1860 7 месяцев назад
Incredible content ! Verry smooth, everything is working perfectly ! (Macbookpro 2021 M1pro, lastest TD version)
@user-fw5yh1cs7e
@user-fw5yh1cs7e 8 месяцев назад
Thanks for publishing it
@roberty9057
@roberty9057 Месяц назад
More than amazing ! Thank you so much
@j.adambrinson8415
@j.adambrinson8415 8 месяцев назад
Fantastic plugin-I think I can pack up my old Leap Motion Controllers now. Question: Is there a way to permanently have h1 be the left hand/h2 as the right? My experience now means that whichever hand is detected first becomes h1, regardless of whether it's my left or right hand. Specifically referring to the normalized data from Output 1 in the hand tracking component. Appreciate any advice on this, as well as the awesome tool.
@shiangchia4337
@shiangchia4337 8 месяцев назад
looks cool! thanks for sharing!
@moonth
@moonth 8 месяцев назад
It's amazing! Thank you
@nightshining5662
@nightshining5662 16 дней назад
Thanks for the plugin! For anyone having issues with the MediaPipe tox loading forever, the last version of 2022 (.35320) works for me on an M1 mac. I just avoided having to upgrade my commercial license for another $300 over this.
@QISUN-ms5rn
@QISUN-ms5rn 10 дней назад
太谢谢惹!!!!!好棒好棒的组件! 😁
@neokortexproductions3311
@neokortexproductions3311 20 дней назад
We appreciate you!
@borjonx
@borjonx 7 месяцев назад
When looking for channel names(like around @20:37 & earlier) you can also just add a SELECT and click the black triangle next to "Channel Name" for a full list.
@denizz_kurt
@denizz_kurt 6 месяцев назад
Awesome tutorial Torin!
@alexfarac873
@alexfarac873 8 месяцев назад
I tried to do the obs set up but for some reason my virtual cam on the media pipe node gives me a black screen. I have tried various resolutions in obs preferences, the input video, my camera's resolution, and 1280x720 but it isn't working. Any recommendations?
@adpokem
@adpokem 8 месяцев назад
How is it that you are able to have both the mediapipe node and video device in run at the same time? I can only ever have one working at a time because my webcam doesn't like it when multiple things try to use it at once. Is there any workaround?
@borjonx
@borjonx 7 месяцев назад
how are you getting the z position of your fingertip from a non-3D camera? ty for sharing, this looks incredible - I've never had such a powerful tox up & running to quickly. Only issue was I had to download the latest TD.
@bradcatlett5215
@bradcatlett5215 5 месяцев назад
Excellent tutorial and plugin. Very well made. I'm trying to apply the face feature points with a 3d model to animate the face according to the user. Any tips on how best to accomplish this? I keep running into blockers or dead ends.
@clintonvanarnam710
@clintonvanarnam710 4 месяца назад
Hey! Thank you for this. So great! Is there a way to upload videos instead of using the webcam?
@lee_sung_studio
@lee_sung_studio 8 месяцев назад
Thank you. 감사합니다.
@mattphillips3024
@mattphillips3024 7 месяцев назад
WOW! incredible
@vxkhwz
@vxkhwz 8 месяцев назад
Hey, thanks! Question: so there is no way to keep up tp date with git pull atm? So I will have to manually download updates?
@spindlewhorls9764
@spindlewhorls9764 2 месяца назад
thank you for your work it's real fun to use this and a lot of possibilities. I have one question: would it be possible to use the mediapipe plugin in td with an Infrared webcam?
@RusticRaver
@RusticRaver 8 месяцев назад
350£ kinect 2 is good for the bin!
@sadiaa.3227
@sadiaa.3227 8 месяцев назад
still useful for pointcloud projects
@doridori8916
@doridori8916 6 месяцев назад
Thanks! i have a question! can i add my hand gestures? (not something that already exists like Closed Fist, Open palm, Point Up ... )
@sebastianosabo
@sebastianosabo 3 месяца назад
Hi guys, first of all I wanted to thank you for this great tool! I am working on a project where, using my hands, I reconstruct images as if I were painting them and I wanted to ask you a question. Is it possible to remove the neck part and keep only the hand part when working using output 1 of the image segmentation 21:33 (the body skin colour one)? In the preview both hands, and part of the arms, are coloured pink/purple but as all the skin is detected the neck, which always remains uncovered, is also detected, ruining the effect. Thanks again and congratulations on everything!
@yoli5222
@yoli5222 18 дней назад
i would be really grateful to understand the spoutcam for windows in depth for the mediapipeline plugin. Thank you
@fuegofinder2015
@fuegofinder2015 7 месяцев назад
Awesome. How is the performance of this vs the kinect?
@jjeon0923
@jjeon0923 7 месяцев назад
Thank you so much! Such a game changer! Is there a way to out put the wire frame in a top? Trying to just out put stick man figure without the source video underneath it. Again, thank you!
@jaspersegers
@jaspersegers 5 месяцев назад
Hey, this is a super great plugin. I have a question though, There's no way to keep a consistent left hand and right hand. Everytime I remove one hand from the frame, the remaining hand becomes Hand_1. This makes it impossible for me to create a controller using 2 hands :(. Does this have an easy fix or is this functionality coming in the future? :)
@alchemist.D
@alchemist.D 8 месяцев назад
nice job
@LealRaulAndres
@LealRaulAndres 6 месяцев назад
Looks amazing, but i could not get it to work, i get an error that resolutions is up to 720p. Tried putting the resolution down but still no working. Working on a MAc pro with sonoma TD 2022.33910
@eric9124
@eric9124 6 месяцев назад
Love the plugin! For some reason the latest version (0.32) shows the facePoints chop as an x,y,z chop (with green, yellow and red) as opposed to the time-slice style chop in the video. Any idea how to convert to the latter? Thanks!
@luckuanfu2022
@luckuanfu2022 6 месяцев назад
u can try to use analyze CHOP i thought
@eric9124
@eric9124 6 месяцев назад
@@luckuanfu2022 Perfect, thank you so much!
@moonmountain
@moonmountain 8 месяцев назад
Thanks so much for this amazing upgrade to working with mediapipe in TD, I was wondering if there is any benefit to using smaller resolutions than 1280 and if the spout method is the only way to achieve that downsizing? With the previous workflow of mediapipe I was able to get good hand coordinates at resolutions as low as 240x135 with an increase of roughly 8-10 fps. Although I am totally unfamiliar with the webbrowserCOMP and how/where it gets the webcam feed. Thanks
@blankensmithing
@blankensmithing 8 месяцев назад
Hey! yeah the models run off a very small resolution feed, so there's no need to use a large video format. The only way to do it at the moment through spout or obs, which would add an additional delay
@Ardsadprm
@Ardsadprm 8 месяцев назад
Hi! I got a problem, only loading and media pipe logo, can't detect the face in the camera, what is the solution?
@user-mv2ps4yi9x
@user-mv2ps4yi9x 7 месяцев назад
cool Torin, thanks a lot 4 da work work work. , do you know if i connect 3 webcam to TD, can i use mediaPipe, using each one to track their onw user's hand?
@blankensmithing
@blankensmithing 7 месяцев назад
Hey, thanks! Yes you can just duplicate the MediaPipe component, make sure to change the Websocket Port and the MediaPipe Port to something different than the original component and you can select a different web cam. You can do this with however many webcams you have. I'd recommend turning off all the models you're not using for speed
@user-uq9cn3wo8e
@user-uq9cn3wo8e 6 месяцев назад
Hey Torin! Thanks for the great video and plugin! I was wondering if there is a way to track multiple people at once. especially with the pose tracking? or is this not possible with MediaPipe?
@blankensmithing
@blankensmithing 6 месяцев назад
Hey at the moment Media Pipe only has pose tracking for 1 person. It does look like multi-person pose is on the way soon though since their web based demo has multiple person detection(those examples are usually head of what they release to the public).
@user-uq9cn3wo8e
@user-uq9cn3wo8e 6 месяцев назад
@@blankensmithing ok great thanks for your reply! Diffrent question: How can i use the Plugin with a higher resolution? I want to use the video output in 1080p but i can not use my Camera Input in a Video Device In TOP when i allready selected it in the MediaPipe Container.
@blankensmithing
@blankensmithing 6 месяцев назад
@@user-uq9cn3wo8e The ML model scales down the webcam size, so using a higher resolution won't give better results. If you need to use a higher res cam you can use Spout Cam, or Syphon w/ OBS to change the resolution of your device and pipe it back into TD as a virtual webcam
@mustaTraceur
@mustaTraceur 4 месяца назад
Hi, I hope you can help me. I just download the mediapipe plugin, so I open the main project MediaPipe TouchDesigner.toe then I Select my main camera and it doesn't show any image, but when i use a videodevice_in it automatically use my main camera. Therefore almost every subsequent node show me some kind of error Im using a macbook air M1, so, the main camera is a facetime camera. Also, when I drag and drop some of the .tox at an empty Touchdesigner project it just doesnt drop at the workspace. Any idea if Im the one thats doing something wrong or someting else happening?
@lilyhsiao9315
@lilyhsiao9315 7 месяцев назад
Thank you so much for your plugin!!! I have a question about the bound box data, there is an "angle" from the mediapipe face bounding box value list, but in my project, it always shows "0".
@blankensmithing
@blankensmithing 7 месяцев назад
Hey! which model are you looking at? Is it the face_detector, object_tracking, face_tracking component?
@lilyhsiao9315
@lilyhsiao9315 7 месяцев назад
​@@blankensmithing thank you for your immediate response! I'm looking for a face detector. I wanted to get the face/head rotation value, and then I saw an "angle" property, but it always shows"0".
@lilyhsiao9315
@lilyhsiao9315 7 месяцев назад
Hi, I just would like to follow up on this if I did some wrong setting. Thank you so much🙏
@daveydonaldson2750
@daveydonaldson2750 8 месяцев назад
what is the best way to use the face mask and replace it with a custom face model, IE make the face track onto a scary ghost or something fun instead of the base White mask. Trying to figure out the best way
@7cd664
@7cd664 4 месяца назад
Did you find a solution?
@eshu1023
@eshu1023 3 месяца назад
Hello.. does this work on TD non-commercial as well? I subscribed to your patreon and downloaded some TD sketches.. none of my webcams seem to work (just blank white page)and I am wondering whether it is the TD version! mediapipe downloaded from your Github has the same issue. My MacOS is 14.0 Sonoma. Thanks in advance.
@blankensmithing
@blankensmithing 3 месяца назад
Hey what version of TD are you using. You don't need a commercial license. It should run on the latest version of TD, but loading into much older version won't work
@eshu1023
@eshu1023 3 месяца назад
@@blankensmithing it's the most recent version. I downloaded it this week.
@BACCAJIMIN
@BACCAJIMIN 8 месяцев назад
hello thank you so much for the plugin!!! I just have a problem of saving the file. Everytime I save my file and exit, when I open the file again, all the work I've put disappears.. if the file is not saved in the same folder of the toxes folder, the work is still there but it says the file is having a trouble loading components(probably looking for the toxes files) How can I solve this? :(
@blankensmithing
@blankensmithing 8 месяцев назад
Hey if you've lost any progress just put the tox files in a folder called toxes that's in the same directory as your .toe file and re-open your .toe file. In the future there's a toggle you can turn off on all the mediapip tox files in the parameter page under Common called "Reload .tox on Start"
@BACCAJIMIN
@BACCAJIMIN 8 месяцев назад
@@blankensmithing omg thank you so much!!! I tried turning the toggle off from mediapipe.tox file and still had the same problem, I had to actually turn off all the toggles from rest of the tox files in the tox folder and convert them to tox files instead of toe files to make it work!! Thank you so much💗💗💗
@federicaintelisano5639
@federicaintelisano5639 Месяц назад
Can you connect cameras like arri?
@kamiusu2360
@kamiusu2360 8 месяцев назад
Hi! I have a problem, when i open a patch all connection from MediaPipe base in disconnected. And i have red circles with cross. How i can connect them again?
@blankensmithing
@blankensmithing 8 месяцев назад
Hey please follow the steps here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Cx4Ellaj6kk.htmlsi=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message
@kamiusu2360
@kamiusu2360 8 месяцев назад
@@blankensmithingThank you for you help! Now it works good on my Apple laptop, but on PC it stucks in loading Media Mapper when i connect usb webcamera.
@kamiusu2360
@kamiusu2360 8 месяцев назад
I resized usb camera resolution in OBS to 720p and it works now! Thank you very much!✨💜
@zablok69
@zablok69 Месяц назад
Hey guys! Anyone knows if there is a workaround to use Mediapipe TD for vertical video / stream?
@amanpathak2338
@amanpathak2338 8 месяцев назад
This is my first time working with a plugin on TD, I followed the steps like you said, and referred to the github page but it doesnt load up media pipe for some reason, I selected my Webcam, checked it as well but MediaPipe doesnt finish loading, what should I do?
@jhahnaaa
@jhahnaaa 8 месяцев назад
I was facing a similar issue. On downloading the current version of TD (2022.33910), it worked well!
@blankensmithing
@blankensmithing 8 месяцев назад
Hey please follow the steps here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Cx4Ellaj6kk.htmlsi=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message. Make sure to list what version of TouchDesigner you're running as well
@amanpathak2338
@amanpathak2338 8 месяцев назад
@@blankensmithing Hi, so I was able to figure out the problem and now it works just fine. Excited to create with it. For anyone facing a similar issue, I used OBS studio instead of spoutcam for the webcam stream [PC] . In the video input of Mediapipe I used OBS Camera (which was scaled to 720p) and thats how it started to work. I created a new scene in OBS and simply added my webcam to the scene. Keeping OBS open I restarted TD and I could see "OBS Camera" as an input in the drop down. Hope this helps anyone facing this issue. Cant wait to get started with this plugin finally:) Thanks to the creators again!!!
@blankensmithing
@blankensmithing 8 месяцев назад
@amanpathak2338 Hey! I'm glad you got it working :) Thanks for the tips so we can pass it along to other people
@user-sq2im8qm8p
@user-sq2im8qm8p 3 месяца назад
Why does mediapipe report an error when I use it?
@AliTanUcer
@AliTanUcer 8 месяцев назад
Another option is NDI in NDI out..
@chrismills8404
@chrismills8404 8 месяцев назад
I feel like your intro gets louder every vid and its a slow prank :D
@blankensmithing
@blankensmithing 8 месяцев назад
haha It's always the same volume, but may have to do this now :)
@yosukeDVJ
@yosukeDVJ 8 месяцев назад
Amazing! This is huge. How can I get Syphon to show up in the cameras dropdown? I'm using someone's proprietary OpenFrameworks Syphon App to get a KinectV2 working on my M1 Mac. I only get the video feeds tho so no CHOP data. Incorporating this mediapipe plugin into my workflow would nearly complete finally having a fully functioning kinectv2 in touchdesigner on apple silicon!
@yosukeDVJ
@yosukeDVJ 8 месяцев назад
never mind! Using ndi.
@NA-cw4pj
@NA-cw4pj 7 месяцев назад
Hi, thank you for this beautiful project, but I am having an issue, why the MediaPipe box is always on Loading mode, and it print an image of Loading. I tried reseting the process but still doesn't work for me. Can you help me? Thank you!
Далее
You are using this Render Setting WRONG! in Blender
3:09
iPhone or Android?😂📱🔥@milanaroller
00:13
Просмотров 2,4 млн
void pointer
5:40
Просмотров 6
Setting up your system for StreamDiffusion TD
15:25
Просмотров 2,4 тыс.
iPhone or Android?😂📱🔥@milanaroller
00:13
Просмотров 2,4 млн