Body and Facial MOCAP | Rokoko + iClone

Подписаться 278 тыс.

Просмотров 39 тыс.

50% 1

In this vlog I combine body MOCAP from Rokoko with iPhone facial MOCAP in a program called iClone. I record the face and body MOCAP at the same time and send it to Unreal Engine / Cine Tracer to be filmed with my HTC Vive based virtual camera.
GDC 2020 Events:
Unofficial Unreal Engine Meet Up - / 604761813648244
GDC 2020 "Cine Tracer" SideFX Houdini Presentation - www.sidefx.com...
Software + Hardware:
Cine Tracer - store.steampow...
Unreal Engine - www.unrealengi...
Rokoko - www.rokoko.com/
Reallusion - www.reallusion...
HTC Vive - www.vive.com/
Redrock Micro - shop.redrockmi...
Cheers,
Matt

Опубликовано:

1 окт 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 84

@CinematographyDatabase 4 года назад

In UE4 the animation was playing 1/3 too fast, so I slowed clips down in post. That is why the motion looks weird. I believe I imported the FBX at the wrong frame rate possibly. I'll update this comment if I figure out the issue.

@bondell1840 4 года назад

update it bro

@thearousedeunuch 4 года назад

Did you find the issue?

@alex0589 4 года назад

You remind me of the very small team that made the game "Hellblade" they had a tiny mocap studio in their conference room and sort of developed their own ways of doing things and made a great looking game for almost nothing compared to other studios. (Their complete production vlogs are on youtube for those who might not know!) You had this up and running in no time, that's impressive. Next up: carbon fiber monopod leg+hockey helmet+ Gaff tape= Performance capture helmet.

@CinematographyDatabase 4 года назад

Alex I’m very familiar with their story and eventually Microsoft acquisition 😂 thanks. I do need phone helmet or mount. Rokoko has one I believe too. The other Technoprops ones are for bigger studios.

@LordJagd 4 года назад

And that was back in 2016! How has the tech evolved since then?

@JQhardge 4 года назад

You are killing it with these VPVlogs! It’s so wild to see how far you’ve come. I remember watching you on twitch everyday back at the start of this. It’s incredibly cool to see all of this progress.

@hellohogo 3 года назад

Could you use vive trackers instead of rokoko suit in iclone for body mocap? even if it's lower quality...

@prankfever8877 2 года назад

i would like a complete motion capture suite with headset like they used in the movie planet of the apes and in ninja turtles . My question is how do i capture my facial expressions if the suite doesn't comes with a headset . Do you have to buy the headset seperately ?

@CHITUS 2 года назад

Hey Matt, great work!

@reallusion 4 года назад

Great work Matt !

@Subrat_Das 3 года назад

Thanks to you for your 3d softwares

@GospelMusicians 4 года назад

I’m new to this, but it seems you have to go from Rokoko to iclone to Unreal Engine? Why are so many tools needed?

@AnimeBadBoi 2 года назад

The normal process is typically longer and require more tools than this

@digimaks 3 года назад

iPhone iPhone...... I USE ANDROID!! Why are there no alternatives!? This sucks.

@oby1 2 года назад

Ouff... the facial capture ain't that great

@JarrenRocks 4 года назад

This is looking fantastic Matt. Something like integrating this into a twitch livestream would be awesome for a unique online experience!

@CinematographyDatabase 4 года назад

yeah this is pretty close to how Vtubers work. You would just live stream it all into UE4 and then stream from a UE4 camera.

@JarrenRocks 4 года назад

@@CinematographyDatabase That makes sense, I'm sure we'll see animated TV shows start adopting this technology in the next few years.

@CinematographyDatabase 4 года назад

@@JarrenRocks they have been using this for a while now, but more and more will use it for sure.

@Sebbir 4 года назад

So the 3D version of Adobe Character Animator? I love it!

@theinfocircles7406 3 года назад

In the introduction the mocap character's mouth looks like he/she is saying "i" the whole time. Hardly any changes to the shape of the mouth when talking. Like a Kermit the Frog puppet.

@theo.jovitch 4 года назад

Hey Matt, cool stuff! Are you planning on creating a keyframe system in Cine Tracer in order to be able to time the different actor performances so that a series of performances from a single actor, as well as several performances from multiple actors can realistically be stitched together? Also being able to keyframe "NPCs" to sync behavior with live performances would be amazing.

@CinematographyDatabase 4 года назад

I'm working on something like that to help sync all the actions together. I want to avoid the traditional "timeline and clips" metaphor however.

@minimallyinvasiveventures 4 года назад

Hey Matt - I noticed that you don't look at the video assist on your virtual camera and look at the monitor instead. It looks like this video assist is pretty expensive part of the kit. Could you use an ipad pro instead? Interested to hear limitations and how you would mount it to the virtual camera rig. The ipad pro would give you additional buttons and inputs - the same as how using a vive controller is better than the puck.

@griffon4163 3 года назад

Ребят смотреть есть в и айфон и мокап костюм и персонаж, объяденил актëра с моделью и получилось классно, но Можно-ли после всего этого проанемировать лицо персонажа?

@quicksolphoto 3 года назад

rokoko have own face capture system. Is it do not work wirh iclone?

@ntygamer20 3 года назад

what model is that vcam? can't search it.

@SpiritscometogetherProductions 4 года назад

Hey Matt, really love the work you do for us all 🎬📽😉 question is there a way where we can import our own 3D characters from iclone 7 into cinetracer? That would be so awesome. Keep up the great work.

@CinematographyDatabase 4 года назад

That’s not currently possible. There is a lot of work to import them and then clean them up in UE4.

@SpiritscometogetherProductions 4 года назад

@@CinematographyDatabase Thanks for your reply. Can't wait for that feature to show up in cinetracer 😎 really love that program 😉

@juanayala4678 4 года назад

I need an iphone

@ErrorJordan1s 3 года назад

are there any benefits to using a newer model iPhone?

@jedi7762 3 года назад

Letting you know I had a live demo with Francesco Turri with Rokoko - gave you a shot out as I told him you were the reason I was there...thx Matt

@CinematographyDatabase 3 года назад

check out my new channel, it's more dedicated to MOCAP and digital character work, I'm back in the Rokoko suit - ru-vid.com/show-UC2gH3UhyNvsyNZwubGTBhIg

@sondosa.9051 3 года назад

Are suit pro capture fingers motion

@mr.shohabbos9002 2 года назад

Where are yu from

@mr.shohabbos9002 2 года назад

Where yu study?

@hiskishow 4 года назад

This is insane!!!

@scottownbey8041 3 года назад

I’m surprised you don’t have the Rokoko Mocap gloves- curious if you have any experience with these in the same pipeline?

@sondosa.9051 4 года назад

Kindly what you recommend to use for hand fingers motion capture and face facial motion I need to capture motion to make avatar in sign language What Software and hardware I should buy

@raulmoreno2499 3 года назад

I wonder if virtual production can be done through a laptop?

@CinematographyDatabase 3 года назад

A modern MSI laptop with a 30** GPU would be an good place to start. But a full PC works better.

@grapicsrz456 3 года назад

Hi thanks for the video but Do you think that with all this equipment it is easy to do fitness animations?

@ToonaciousD 4 года назад

What the link to the virtual camera hardware you use? Thanks.

@wittdesigns 4 года назад

Hey Matt, great work! There is an update to cinetracer that allows to import fbx scenes, custom character, etc? Keep up strong!

@keithtam8859 3 года назад

if I have a pc would the iphone works with it?? or do I need them both be apple products?? thanks

@iHeartDom 4 года назад

Do you need the vcam to put your digital character into the real world like that or can it technically be any camera?

@mrtobycook 4 года назад

This is great, Matt! This is the workflow I'm using too and yeah, there's lots of hidden "gotchas" with the process if you want to make it all look perfect. :-))

@AAa-tn5rz 4 года назад

Could we do this with extra Vive trackers and valve index controllers to get full body with finger tracking? If we don't have rokoko? And iPhone 10 face tracking to record full body, fingers and face tracking?

@CinematographyDatabase 4 года назад

Yes, but you would need to write your own IK elbow solver and translate the hand/finger tracking yourself in Unreal Engine and also write/maintain the iPhone iOS ARkit app that runs on the phones. Unless you are asking if it's possible in Cine Tracer, then I would have to do all of those things lol.

@joelarvidsson 4 года назад

Love this video. Seams to work with out any large hiccups.

@belbozs 4 года назад

You should make a helmet rig with your iPhone

@woodrowjang 4 года назад

how big are the file sizes for animation data? say for something short like your intro speech?

@bushellmediainc3700 4 года назад

We rendered a 2 min animation from iClone to unreal and it was 72 gig we used handbrake to compress

@justinkringstad4167 4 года назад

why don't the hands move?

@FINALFORMSTUDIOS 4 года назад

I see that you have the Ninja V on the end there for your virtual cam setup. Are you able to see the Unreal engine display on there or what is its purpose?

@Jsfilmz 4 года назад

he records from ue4 straight to the ninja

@ramiru3264 4 года назад

How much is the mocap suit

@wongj3279 4 года назад

So the real-time live avatar streaming works right? you have plan doing this?

@CinematographyDatabase 4 года назад

yeah I'll show some real time LiveLink into UE4

@patriotaRBC 4 года назад

No hands motion cap ?

@PiPiCatStudio 4 года назад

no fingers capture, too sad

@ranjithvenkat4410 4 года назад

i want cine tracer for Android

@noahstudio3717 4 года назад

You said Live Face is free, I just checked it . It is not...it is $399.

@marcuscisneros8528 4 года назад

Ashu Ka the iPhone app is free but it’s pretty much useless without that 400$ plugin for the desktop app

@ramiru3264 4 года назад

@@marcuscisneros8528 so you have to pay for the plug-in and the software? Or just the software and the plug-in is already there?

@Ranger7Studios 4 года назад

Amazing opening.

@Myshowbro 4 года назад

My question for you is Are there specific limitations to why Android is not supported it would have helped if you went into more detail

@CinematographyDatabase 4 года назад

It's technically possible, but no developers have supported Android yet for facial MOCAP. Apple has a very robust depth sensor based facial anchor system and the API to use it in third party apps is pretty straight forward.

@Myshowbro 4 года назад

@@CinematographyDatabase thank you for the timely response it was greatly appreciated

@ZigUncut Год назад

You could use a gopro mounted to a helmet and use Faceware on a PC. The results are amazing. In particular the eyes which is something that the iphone isn't so good at.

@VicVegaTW 4 года назад

You are a one man army

@goodcitizen17th34 4 года назад

Awesome

@SharpDesign 4 года назад

Face is cool but the flat hands are throwing me off a bit.

@CinematographyDatabase 4 года назад

Sharp Design yeah I hope to have some hand tracking hardware in the future

@SharpDesign 4 года назад

@@CinematographyDatabase awesome.

@ВладЛеонтьев 4 года назад

а где пальцы?

@overdev1993 4 года назад

the facial animations look really crappy perfect body motion capture + crappy facial animations is a really bad combination

@CinematographyDatabase 4 года назад

There is still some tweaking on my end to get the facial animations better. The phone wasn't placed in an ideal position and you have to learn how to puppet it correctly. There are only so many blend shapes that it can match to.

@alex0589 4 года назад

That's because youre used to games that have animators work for 2 years on cutscenes facial animation ahah, this is just a proof of concept

@KRGraphicsCG 4 года назад

I often keep noticing this with the iPhone... you have to exaggerate your facial motions to get everything looking right. It's why I started creating my own facial pipeline with custom hardware.

@KRGraphicsCG 4 года назад

@@CinematographyDatabase It is very hard. It's why I prefer the precision of marker based facial capture... I can capture all kinds of nuances in the face with this method. I hope to learn more from you

@KRGraphicsCG 4 года назад

Also the frame rate in the iClone face cap is 30FPS... you need at least 60 FPS or greater to get VERY good facial animation