Real-Time Motion Capture with a Cheap Webcam (or iPhone): TDPT - Unreal Engine 5 Tutorial

Подписаться 1,2 тыс.

Просмотров 34 тыс.

50% 1

Learn real-time motion capture using a cheap webcam or iPhone with ThreeDPoseTracker (TDPT) and Unreal Engine 5. Follow my step-by-step tutorial video to discover tips and tricks for capturing motion in real time. Skip the hassle of retargeting or working with other types of skeletons and instead, use Unreal Engine's Metahumans. You'll be amazed at how effortlessly you can achieve it!
I'd like to express my gratitude to Digital Standard Co. and Developer HAL9HARUKU for releasing their programs for free, enabling us to perform real-time motion capture. Many thanks!
● Download links:
1 - ThreeDPoseTracker (TDPT)
(Click the "Free Download, 0 JPY" button. The website may ask you to sign up)
digital-standard.booth.pm/ite...
2- VMCtoMOP
(Scroll down the page a bit and download the file named "VMCtoMOP.zip" (31.4 MB) located in the "Assets" tab)
github.com/HAL9HARUKU/VMCtoMO...
3- MopUE4
(Go to the "Code" tab and click on the "Download ZIP" button)
github.com/HAL9HARUKU/MopUE4
● Learn more:
TDPT Windows:
digital-standard.com/tdpt-win...
TDPT iOS:
digital-standard.com/tdptios_en/
Developer Yukihiko's Article:
qiita.com/yukihiko_a/items/23...
● The secret sauce of my video's musical awesomeness:
Background music:
www.epidemicsound.com/track/o...
The funky beats accompanying Metahuman's mind-blowing dance moves:
www.epidemicsound.com/track/n...
Use this link to register for Epidemic Sound, supporting my annual subscription and allowing you to enjoy good music in my videos.
share.epidemicsound.com/ubem7i

Опубликовано:

20 май 2023

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 157

@duz_gen 6 месяцев назад

Hey guys, quick update after 7 months: The current Unreal Engine version is 5.3.2, and the workflow here doesn't support above 5.0.3. Updates not only apply to Unreal Engine but also to Metahuman versions, possibly causing issues with older Unreal Engine versions when using updated Metahumans. I tried reaching out to the VMCtoMOP plugin developer but got no response. Maybe he missed my message, I don't know. Honestly, I love this workflow and the video-it's a good, free option, but it's slowly losing its freshness. While figuring out how to adapt this workflow to the latest Unreal Engine versions, I found another software called Dollars MONO. It offers real-time, full-body (face, fingers, body) Metahuman mocap, so it's an all-in-one solution, no extra programs needed but you can also combine it with Live Link if you want. The catch is it's not free; there's a one-time fee of $99, but you can try it out with a trial version to see if it's meets your expectations. Just to clarify, I don't have any deals like an affiliate link or any involvement in receiving a percentage of the sale with the company. I got access for free, which is the reason for the 'paid promotion' label. Check out my video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ayTm856fDBQ.html By the way, this was my first tutorial video, and the response was beyond my expectations. Thanks a bunch for that!

@leeanne333 Год назад

Great vid, great to meet you! Hope to see many more of your videos!

@duz_gen Год назад

Thank you, stay tuned for more exciting content! I’m glad you enjoyed the video and it was a pleasure to meet you too Gabriel!

@kopfjogger 17 дней назад

This is unbelievable cool!

@wikingodelaruta2083 3 месяца назад

sos un genio!!!! gracias desde argentina

@miojosmole 7 месяцев назад

Good luck on your channel!!

@wagnemo Год назад

brilliant, Thanks for sharing

@castleliao7357 8 месяцев назад

Very useful, thank you!

@YosiSarangaOfficial Год назад

Thx Man, big love!

@HanmaHeiro Год назад

Here's your 271st sub! Thanks for the video

@duz_gen Год назад

Thanks for the sub, welcome! 🙌

@mestresan9141 Год назад

Muito bom, obrigado pelo conteúdo

@SuperlativeCG Год назад

Che figata!

@techarchsefa Год назад

That's great thank you

@abo2arabarab418 Год назад

Thnx Sir

@indiestudio23 10 месяцев назад

New subscriber added respect to you brother ❤

@duz_gen 9 месяцев назад

Thanks and welcome!

@R00kTruth 7 месяцев назад

I'll program my own thanks...

@ronioclarenzo6137 2 месяца назад

Excellent tutorial! Thanks a lot! Can you make a video on how you can create a multi-person online game with this?

@jackrobin9829 8 месяцев назад

Good video bro, love from China 😀 hope to see more videos from you

@duz_gen 6 месяцев назад

Thank you, there will be more videos! 👨‍💻

@xfyin3969 10 месяцев назад

Thank you very much, very detailed, I have subscribed. If possible, I hope to see you record an installment of how to capture with custom characters next time!

@duz_gen 9 месяцев назад

If you're asking about custom characters other than metahuman, yes, you can definitely do real-time mocap with them as long as the skeleton matches.

@itarpas 2 месяца назад

Hey! thank you for this video, it's been really useful for some projects. But I realized that the head is not tracked. I mean, the tracker does it but the MetaHuman doesn't seem to recieve that info. I'm not able to identify if in your case it's the same or not. Are you having the same issue or is something that only happened to me?

@Conker2350 10 месяцев назад

subbed good channel

@notgesund Год назад

Great video. If I use this method to livestream, is it better to have a streaming pc and a gaming pc separately?

@duz_gen 11 месяцев назад

Yes, it would definitely be better because this method puts a load on the graphics card.

@mt_gox 9 месяцев назад

Wow! Vlad The Impaler sure knows his UE5 stuff!

@duz_gen 9 месяцев назад

Do I really look like Vlad the Impaler? 😄 heard that he's vicious but I'm not too familiar with his history

@jocortzam 7 месяцев назад

5.0.3 doesnt want to load the metahumans only 5.3 , what can i do ? it says they are newer......

@cosmovisionbrasiltt6023 Год назад

Very good content, I learned a lot from your video, you gained another subscriber to your channel, you certainly didn't talk to yourself! Soon more subscribers will appear, anxiously waiting for the next video. Success and I will follow whenever I post new content on your channel. Hugs from São Paulo Brazil.

@duz_gen Год назад

Wow, thank you so much for your kind words buddy! It's encouraging to know that my work is reaching and helping people like you. Your support means a lot to me, and I'm excited to have you as part of our growing community. Agradeço de coração!

@cosmovisionbrasiltt6023 Год назад

@@duz_gen Pode contar com isso meu amigo! Já me inscrevi e agora sou mais um dos seguidores do teu canal, e sempre interessante assuntos que envolve essa área da animação e produção virtual com a unreal engine, e fantástico!

@Thefirehunters Год назад

@@duz_gen❤

@UE-5 Год назад

Good Video We do notice the little guys ;)

@duz_gen Год назад

Thank you! 🙌

@tongzhouwarrior 8 месяцев назад

Thank you for the nice video. Is it possible to use a mirrorless camera like Sony A6000? I currently have 2 cameras. Lumix G7 and A6000. I also have a Samsung Note 10+ Could I use any one of these with this technique?

@duz_gen 6 месяцев назад

Absolutely, you can use it. Just connect your camera to your PC and use OBS's virtual camera function. If you have a capture card, even better. If you plan to use your smartphone I recommend the IVCam app

@TheDailyMemesShow 10 месяцев назад

Can this work with a multicam rig, exported to any 3d animation software, and what about facial capture with high accuracy? Thanks!

@duz_gen 10 месяцев назад

Not sure about the multicam rig but in the desktop version, there's an option to save motion data in BVH format. Just click on the 'Record BVH' button in the menu. On the iPhone version you can save in both BVH and VMD formats and export them as you like. Both formats can be converted to FBX, so you can easily import these FBX files into most 3D software

@Shji123 6 месяцев назад

cant find mopr please help me

@amrattia84 10 месяцев назад

Awesome! Could i apply the same concept but not with (TDPT), I wanaa to use it with (VMC) virtual motion capture app?? is it goes right with the same workflow? Thanks.

@duz_gen 9 месяцев назад

That should actually work because Unreal doesn't directly interact with TDPT, it connects with vmctomop and as long as you can run vmctomop you should be able to use any software with Unreal. However, keep in mind that the software you put in place of TDPT may have different settings and you will need to configure them properly.

@amrattia84 9 месяцев назад

Thanks, i have tried it and it works well🙂. @@duz_gen

@AdityaSharma-ph6qi 9 месяцев назад

Thank you so much for the plugins but, after I dragged my mop receiver into the scene it says bad on the blueprint icon and nothing is working , can you help me out?

@duz_gen 9 месяцев назад

Still dealing with that issue or did you manage to fix it?

@harriet.blend. 9 месяцев назад

I have the same issue :(@@duz_gen

@madhuranagle5454 11 месяцев назад

Hey, any idea how I can assign the mop receiver to a game pawn?

@duz_gen 11 месяцев назад

Hi, what's your goal with using the mop receiver in that way? Its only function is simply to keep the program running, nothing else.

@brianwilson7934 11 месяцев назад

Going to try this out with XR Animator to see if I can get hand tracking working along with the IOS live link face

@brianwilson7934 11 месяцев назад

Oof no dice on getting hand tracking working but if anyone knows how to get it working I'd be very interested

@duz_gen 11 месяцев назад

I hadn't heard of XR Animator until now. I just checked it out and it'd be great if there's a way to make it fully work with Unreal Engine. I'll do some research and see if it's possible@@brianwilson7934

@acecita 6 месяцев назад

The program ThreeDposeTracker has for hand tracking from what I see, but how do we detect it in real time?

@duz_gen 6 месяцев назад

It might be possible to achieve this by tweaking the metahuman finger skeletons and then retargeting them. I have a basic idea of how it might work, but I'm not sure about the exact steps to get it done. If you're looking for an easy-to-use full-body mocap system using just a webcam, be sure to check out my latest video. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ayTm856fDBQ.html

@entropicvibe 9 месяцев назад

Do you know a way to make the fingers movement work?

@duz_gen 9 месяцев назад

Recently replied to an e-mail about this: Wrist movements can be transferred quite easily. As for finger data, I believe it might be possible to achieve this by tweaking the Metahuman finger skeletons and then retargeting them. However, I should clarify that I'm not an expert on retargeting. I have a general idea of how it could work, but I don't know the exact steps to accomplish it. That said, the general idea would be to align the finger bone structure of your motion capture data with that of the Metahuman model in Unreal Engine. Once the skeletons are aligned, you could then retarget the motion capture data to the Metahuman model. This would involve mapping each bone in your motion capture data to its corresponding bone in the Metahuman model.

@martinmenso6671 5 месяцев назад

Would it be possible to get the tracked webcam head location from the Evaluate LiveLink node (or any other way to get the data) to track the head location 2d to world location in Unreal Engine. What I am trying to achieve is to have a virtual avatar to always look towards the person on the webcam.

@duz_gen 4 месяца назад

So basically, are you thinking about having the camera follow your avatar's eyes all the time, making sure they always maintain eye contact with the camera?

@martinmenso6671 4 месяца назад

@@duz_gen I found a way :-)

@tapikoBlends 10 месяцев назад

is it true that you can do real time mocap if you hav only xbox 360 kinect?

@duz_gen 9 месяцев назад

No, that's not true. I'm using just a basic $20 webcam here and can still do real-time mocap. The Kinect can also be used for real-time mocap and I think it will give better results.

@kayametatechnology9894 10 месяцев назад

hello, please add iphone face track and your body mocap app in same time tracking course. so we can complete all system body and face

@duz_gen 9 месяцев назад

planning to cover that but for now you can follow the body mocap steps in this video and then set up live link as usual. Both should work seamlessly

@377omkar Год назад

how we can pair this with live link to get both, also live link tutorial

@duz_gen Год назад

This workflow (TDTP + Unreal Engine) allows you to set up a live link connection. It's a standard live link setup with no differences, and it works as expected. While I don't have a tutorial video yet, I may create one soon

@bkunal88 Год назад

it's been like a month now, is it workable in UE 5.2 now ? if not, is it coming soon ?

@duz_gen Год назад

UE 5.2 is not supported yet but if there is an update I'll comment here and let you know.

@Goofy_Music Год назад

Do you have to have premium?

@Goofy_Music Год назад

Could I make possibably export the animation to something like Sourcefilm maker?

@duz_gen Год назад

In the desktop version, you have the option to save motion data in the BVH format. Just look for the 'Record BVH' button in the menu. On the iPhone version, you have even more flexibility as you can save motion data in both BVH and VMD formats and export them as needed. I googled it, and it seems that both BVH and VMD formats can be converted to the FBX format. If Source Filmmaker supports the BVH or FBX format, you should try, i don't see any reason why it wouldn't work.

@Goofy_Music Год назад

@@duz_gen Thank you, also could I still do animation like putting in SFM or unreal without premium?

@duz_gen Год назад

@@Goofy_Music The desktop version is completely free, and there is no premium. However, if you are using the iPhone version, you will need a premium subscription to export motion data.

@user-pg9sl1ql9x Год назад

It would be cool if result can be exported to blender or just some 3d format like fbx

@duz_gen Год назад

@TheDailyMemesShow 10 месяцев назад

Blender and Unreal engine?

@bangsaenairesearchcenter8885 Год назад

keep working....

@enPassantWarrior9047 3 месяца назад

what about facial animation but not just facial animation but facial animation+ body mocap from the video

@duz_gen 3 месяца назад

Take a look at my latest video at ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ayTm856fDB.html. You can use livelink for facial mocap and dollars mocap for body animations. What you mentioned is possible

@elarcadenoah9000 11 месяцев назад

what about if i paint some green dots in my wall ,can i mockup a video in that wall like if i am in a news transmition youtube chanel in real time or news weather in real time,can unreal engine track green dots in the wall so i dont have to pint the entire wall in green color?

@duz_gen 11 месяцев назад

What we're talking about here is 'real-time compositing.' A few green dots won't be enough, so you're going to need a full green screen in the background. Instead of painting your wall, you could consider buying a portable green screen. I have one myself, and it's really handy. You should check it out. (Neewer collapsible green screen)

@elarcadenoah9000 11 месяцев назад

@@duz_gen BUT WHY OR HOW THEY USE DOTS IN THE BODY AND WORKS? ISNT THAT POSSIBLE IN UNREAL ENGINE?

@duz_gen 11 месяцев назад

Those dots are usually used for "tracking". Something different than what you're planning. Painting the wall or putting up a green screen behind is a completely traditional method but they will 100% work. Nowadays, there are a lot of AI tools coming out, maybe you can find what you're looking for in different tools.

@jieli5495 5 месяцев назад

非常感谢，很详细。但是我有一个问题，我想角色替换自己的小孩子模型。该怎么弄？

@duz_gen 5 месяцев назад

Glad you liked it! You need to learn how to use VRM models in Unreal Engine. Please search for 'How to use VRM models in Unreal Engine'

@abo2arabarab418 Год назад

Whats the best motion capture with iclone to ue5…?

@duz_gen Год назад

sorry, don't know much about iClone

@vivekanandhan555 Год назад

Hi, I couldn't find "mopr" after compiling and saving. Any idea?

@duz_gen Год назад

Hi, make sure that you search 'mopr' in 'ALL' folders, not just in 'content'

@vivekanandhan555 Год назад

@@duz_gen Hi, I did search in "All" parent folder, I'm using UE5.1.1 and it doesen't show plugin folders. So I tried again by copying the BP files from the plugin folders into the content folder and now I can drag drop the Mop Receiver into the world. But after setting the port, in the metahuman BP, I couldn't find the Mop Receiver. It's simply showing None. Also tried restarting.

@vivekanandhan555 Год назад

probably I will test with 5.0

@duz_gen Год назад

@@vivekanandhan555 sorry but UE 5.1.1 isn't compatible with mop plugin, only 5.0.x versions work, mentioned in 1:40. Convert your project to version 5.0.3

@vivekanandhan555 Год назад

@@duz_gen Oh you did mentioned it! sorry my bad. Thanks!

@ellysavatia Год назад

Can it capture detail, ie finger movement?

@duz_gen Год назад

TDPT can capture finger movements but it can't be transferred to Unreal Engine. However, with some adjustments to the metahuman skeleton, it might be possible.

@aubreymorgan9763 10 месяцев назад

can this be used for live tracking?

@duz_gen 9 месяцев назад

This is for mocap, I don't think tracking would be possible.

@TigranAivazian Год назад

On 06:56 you find BP_MopReceiver, but in my case there is none. I am using Unreal Engine 5.2 -- maybe that is the reason??

@duz_gen Год назад

Hi, the Mop plugin doesn't work with UE 5.2 (take a look at 1:40) You'll need to downgrade your project to 5.0.x version. In the tutorial, I'm using 5.0.3. It's just a possibility, but if this video receives enough views, the developer might update these plugins to versions 5.1 and 5.2, which would be highly pleasing to everyone.

@TigranAivazian Год назад

@@duz_gen Thank you. Yes, I noticed the UE 5.0.3 requirement, but still hoped that it might work with 5.2 :)

@itsjustview5759 10 месяцев назад

@@TigranAivazianhey if you were still wondering it does work you have to enable show plugin content

@thegoaks 10 месяцев назад

@@itsjustview5759 Thank you!!!!! I was struggling to find the file even in 5.0. CHEERS!

@newchallengers9420 8 месяцев назад

@@thegoaks how TF did u find it??? I'm on 5.0.3!!!

@CanePazzoHD 11 месяцев назад

hi! is now avaiable for 5.2 version?

@duz_gen 11 месяцев назад

Not yet but if there is an update I'll comment here and let you know.

@Ben_CY123 Год назад

If should be cool if combine live link face app on another phone, low cost full body mocap solution.

@duz_gen Год назад

This combination is possible I've used it in a project and it worked well.

@luniared9315 9 месяцев назад

Sry to disturb but does anyone know if it is available for ue5.2?

@duz_gen 9 месяцев назад

UE 5.1, 5.2, and 5.3 aren't compatible yet

@daisylassie280 Год назад

Yes thats much better than sitting in a room and talking to yourself😂😂😂

@Deezeone Год назад

Can it work with kinect??

@duz_gen Год назад

Not sure, I don't know much about Kinect

@ugurtok649 Год назад

başarılar kardeşim. türkçesinide bekliyoruz.

@duz_gen Год назад

Teşekkürler, altyazı ekleyeceğim yakın zamanda

@Rizwan-Ashraf 10 месяцев назад

Wow but Nothing for mac is upsetting! Rokoko is expensive :(

@duz_gen 9 месяцев назад

sorry, maybe a different version for mac will be released in the future.

@vuhung4934 Год назад

On 06:56 you find BP_MopReceiver, but in my case there is none. when i copy BP_Mopreceiver file to Content folder i can find it but in last step at 7:22 i can't find it. I am using Unreal Engine 5.0.3.

@duz_gen Год назад

Make sure that you search for 'mopr' in 'ALL' folders, not just in 'content'

@vuhung4934 Год назад

@@duz_gen yes, i search in All folder. I did all the same steps as you but when it comes to adding mopreceiver file, this file doesn't appear

@duz_gen Год назад

@@vuhung4934 After reading your comment, I started a new project and it worked as expected. In the final step, I select the metahuman and search for 'mop' in the details section. You might have missed a tiny detail. I'd be happy to help, so feel free to email me. I can send you this project as a reference

@dennismeelis3452 Год назад

In the content browser click the settings menu and make sure "Show Plugin Content" is checked. This fixed it for me

@newchallengers9420 8 месяцев назад

@@dennismeelis3452 thankkkkkkkyyyyyyooooouuuuu!!!!!!!!!

@newchallengers9420 8 месяцев назад

THERES NO BP_MOPRECIEVER IN \ALL\ WTH, (UE5.3.1)

@newchallengers9420 8 месяцев назад

ALSO DOESNT EXIST IN 5.0.3 EITHER, WTH??????

@ripulse1839 10 месяцев назад

I can't connect the metahuman MOP to the Mop receiver :

@ripulse1839 10 месяцев назад

its all red out. Im so close to the last step

@duz_gen 10 месяцев назад

Can't find mop receiver?

@ripulse1839 10 месяцев назад

@@duz_genyes. What I did was I manual placed the BP_MopReceiver.uasset to the content folder. But as I drag the MOPreciever to the MOP of the metahuman it reds out

@ripulse1839 10 месяцев назад

@@duz_gen drive.google.com/file/d/14DqatoKwezElX4j9m2AsTsfcV9Zuux4t/view?usp=sharing I'm stuck on this part

@ripulse1839 10 месяцев назад

since typing the mopr wont appear even if I click at the topmost of the folder. I did downgraded to 5.0.3 and changed my metahuman on 5.0 version

@danielhyunlim4721 Год назад

where can i find 5.0.3?

@danielhyunlim4721 Год назад

does it work for 5.2 now ?

@duz_gen Год назад

Open the Epic Games Launcher, find the Unreal Engine tab, click on the plus sign next to 'Engine Versions.' From there, you can install the version you prefer.

@duz_gen Год назад

Currently, the 5.1 and 5.2 versions don't work, but they might become available soon.

@im_Dafox Год назад

Why do some of the best/useful pieces of software always look like they come from a notepad tutorial on how they can help pirate your friends MSN account if you enter your credentials Great tutorial btw, shame it doesn't support 5.1/2 but only a matter of time I hope!

@duz_gen Год назад

Your comment made me laugh 😂 but I don't think it's that bad. However, there is room for improvement in the interface. Since these programs are provided for free, we can assume they don't make much profit. Still, let's give thanks to the developers who make this workflow possible!

@oebleh 6 месяцев назад

Works in 5.3.2 @@duz_gen

@jefff2127 4 месяца назад

can it capture lip sync

@duz_gen 4 месяца назад

nope, check this: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ayTm856fDBQ.html

@bigblack2295 9 месяцев назад

ses biraz değişik geliyor boğazdaki mikrofon sanki gırtlağın sesini değilde hava akışını daha fazla alıyor gibi ses havalandırmadan geliyor gibi sürekli hıhhı hıııhıı hıııhıı tarzı

@shivangipriya4153 5 месяцев назад

Thank you, but how I do this for VRM model ?? any video because no one explain this .. can please

@duz_gen 5 месяцев назад

You can do it with TDPT but it's kinda complex. There's simpler software options out there ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Sc2sk1hXN9c.html And for the software in this vid, I've got a detailed tutorial. It's for metahuman but it'll give you the idea. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ayTm856fDBQ.html

@Shji123 6 месяцев назад

Won't work for me😭😭😭😭😭😭

@duz_gen 6 месяцев назад

which version of unreal are you using?