Using an iPhone as a tracker for Virtual Production 

Not arf!!!

Be sure to check out CamTrackAR from Hit Film.

Great to hear! If you experiment with this, I'd love to hear your findings.

As I’ve said in some other comments here, this method might not be the best approach going forward. It seems like there’s at least one app being developed to handle the tracking of the camera with a phone much better (filmengineers dot com had something in beta which seems like it could be the perfect solution).

@@hyperbolicfilms there is no video or something about these "filmengineer . com " solution in this field as far as I search there is nothing else but only their page

@@hyperbolicfilms Thanks a lot for this, its been one year now, do you know if new latest tracking apps have released for this yet?

@@anirudhsays1534 I have not been following closely. There were a few promising apps like byOwls and Filmengineers, but I have not seen anyone’s result using those. I have been thinking of trying this again a year later to see what’s possible now. Hopefully I’ll have time before Christmas.

Thanks this helped with the edges!!!

probablemente no sea bueno para la producción. solo un experimento

Thanks a lot!

A lot of this information may not apply in more recent versions of Unreal Engine, as a lot has changed in almost 2 years.

Blind Gamer Good luck!

You should consider that this wasn’t far from the limit of where I got with this approach, and I don’t know that it is really viable for production of any kind. In some of the comments, I’ve listed some of the companies making plugins to make the phone tracking approach a little more reliable, but I don’t know that any of those are production-ready over a year later.

@@hyperbolicfilms yeah, no doubt, I was just looking for simple stuff around the house - mainly to have the knowledge of it. I also didn't feel like spending $600+ on sensors and such just to mess with it a bit when I have an iphone and an ipad and a dslr laying around. It's a good start to bounce off of.

I'm not sure I'll get to the projection or LED wall stage of things, just based on the limitation of the space I have in my apartment. I just moved to a larger space, but still there's no real good space for a projection based on the layout. If I can get into someone's studio for a while, I'd definitely be interested in playing with it, but I've moved more towards animation in Unreal rather than live action/CG background integration because it's doable for less money. I'd suggest looking into Ian Fursa's work (ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-_FC1lvhoPRc.html). He posts a lot of his experiments on Matt Workman's Facebook group and he's definitely shown that you can have some success with a low cost projector and a Vive tracker, so probably around $1000 for the hardware.

@@hyperbolicfilms Thank you for the link and not sure either way will be easy on our pockets!

I uploaded a 4K file, so I'm not sure if RU-vid is still crunching that or what.

Improved lighting, 25 fps, RTX 2080, and a better camera will fix this.

You can now do this much easier than I was trying in Aximmetry with their new Eye app

Thank! I haven’t been able to find that, but i have heard that there should be something to compensate.

I'm trying to figure that out myself, if I get it going in the next few weeks I'll post a step-by-step as well. I assume it has something to do with nDisplay still. I was thinking that using a green screen would be good for wide shots, but maybe getting an 80 inch TV would work for close-ups and like toy commercial/product shot type things.

zoombapup I think there are too many limitations to this approach. Shortly after I recorded this, I realized that doing any sort of severe Dutch angle on the camera would cause the iPhone to switch into portrait mode, which makes the traffic unreliable. My best results were in capturing the camera movement data, recording a take of the live action plate in camera, and combining the live action and CG element in After Effects. It’s definitely not nearly as accurate as a Vive tracker would be, but I think it can work for some shots.

@@hyperbolicfilms Good point. No way to lock the camera in landscape mode? I'm using a vive tracker and a single base station and that approach works pretty well. I think these virtual cameras are probably best for doing entirely virtual shots though. I've hooked up a vive tracker based shoulder rig with a virtual camera setup and that works well (made some custom controls for an xbox PC controller). I'm also doing some work on controlling take recorder via an app on my surface pro so I can do different virtual setups and pull focus and play with the virtual cameras.

@@zoombapup I've mostly moved on to playing with mocap and animated characters for my own work. I think Epic could make the iPhone do what I was hoping it could with some adjustments to the Unreal Remote app, but considering the situation now with Apple, I doubt it would happen. It's frustrating because it is close to being a good solution for mixed live/virtual productions, but just not enough in the app.

hyperbolic films the virtual camera spawns a new camera instead of using yours.

How should I go about adding the buffer? Just using an iphone but the tracking is coming later than my video feed

I haven’t had much of a chance to revisit this workflow since the summer as I’ve been back at work the last few months l, so I don’t have a good answer for you. The URemote does seem to pick its starting point and there’s no way to change much. It’s a very blunt tool, and I hope they add in more control at some point.

@@hyperbolicfilms thanks. No worries. If I work it out I’ll let u know. Cheers

same issue.

In theory yes, but this method has broken since Epic changed things in more recent versions of UE. Their new mobile app doesn't work the same way and doesn't seem to work with many iPhones. I've tested it on my iPhone 11 and a new iPad and they won't connect with Unreal.

@@hyperbolicfilms that's sad. Thank you for your reply!

Glad you like it!

It should record any movement, except Dutch angles. Because of the iPhone’s inability to lock into landscape mode, tilting into a Dutch angle causes the Unreal Remote app some problems.

@@hyperbolicfilms thanks for this valuable information.

I don’t know. The approach that I was taking after this was to record the green screen in camera, render my CG background in Unreal, and composite in After Effects. As of three days ago, there is an app in beta testing for Android and IOS that makes this approach moot. It apparently does the phone tracking and allows you to offset the position of the data to account for where your camera is. I’m not a beta tester unfortunately so I can’t say how good it works, but it seems like it’ll be a great tool for indie filmmakers if it works.

John Gabriel The mount is a simple tension-based phone holder attached to the camera by a cold shoe to 1/4 20 mount. This is a Manfrotto phone holder, so was about $20, but I have a few $2 ones that would do just as well.

Hi, a teacher at the college where I took a virtual production course this summer tried to recreate it with the latest version of the software, and he determined that there was no good way to get sync. I still haven't found a good, reliable, affordable way to do virtual production. I had tried Assimilate LiveFX, which is a very good live compositing program that says it supports tracking from phones or the affordable Intel RealSense camera, but even using two fairly powerful computers with multiple SDI I/O cards and timecode, it was still out of sync. I could record the Unreal background and the video in the camera and sync the two and use LiveFX for a video preview though. There is a new very affordable tracker that hit the market now (has similar packaging/marketing images to the ReTracker Bliss, but I can't recall its name). It might be the game changing device, but still doesn't have a lot of software support. The fastest/most affordable way to do any of this is using the CamTrackAR app on iOs, but it has its limitations and isn't real time.

@@hyperbolicfilms thanks for the very detailed answer! I really appreciate it. 🙏🏼

No idea. I think this would take Epic making a big update to their app, which seems unlikely.

These are the places that I'm stuck, and I wonder if it's possible to get past that hurdle. Seeing others worth with the Vive tracker (like Greg Corson's great project and tutorials on his RU-vid channel), there are specific steps to follow to do the alignment to get everything accurate. I don't know that it's possible to get perfect alignment of lens and offset with this iPhone method using the virtual camera. There are a lot of aspects of how the virtual camera works that are a mystery to me, such as what direction it will start when I begin my scene. It will start in the position of the Cine Camera in the scene, but not in the direction that the Cine Camera is pointed. I don't know if this is a limitation of my knowledge, or a limitation of the virtual camera and remote control app. There's a fair bit of documentation on how to use the virtual camera, but not a lot on how it works.

Also sorry for the slow reply, I haven't been seeing notifications for new comments!

hyperbolic films did u ever work out how to solve the direction the iPhone/iPad view starts at? Even if I set waypoints in the app my camera goes to the point but again at an annoying offset angle

@@tmaintv No, it seems to be a limitation of the app. I think it creates a camera specifically for the virtual camera. I really think the Byowls Android app will surpass all this because they are actively developing it and adding new features. Epic hasn't done much to update the app, and the documentation is so minimal that I was guessing at a lot of things when figuring this tutorial out.

I think things may have changed in more recent versions of Unreal. This was made in 4.23 I believe. There is a new version of the virtual camera app, but I have not been able to get Unreal LiveLink to see my phone with it.

Thanks very much again for this tutorial!

I think that has to do with the way Unreal Engine processes the two elements. It's taking more than realtime to render the CG element, but is recording the video in real time. So the final video becomes fastforward. In my later experiments, I recorded a take to get the virtual camera motion recorded, rendered out the CG element, recorded the live action in the physical camera, and then composited in After Effects. I think there's a good chance that the Byowls Android plugin will be better for this eventually because they are actively developing features still. Epic put out the Remote, and haven't added anything new since the initial release.

Thanks for the info. I have not had the opportunity to dive into this much since I made this tutorial. I’m going to start using Aximmetry in the near future to see how well that does, partially because of the keyer and because it already has the input and output side of things handled.

@@hyperbolicfilms Would be great to share your experience again :)

Will Huff You do get both translation and rotation with this technique. There are a number of problems with the accuracy though that I haven’t sorted out. The Unreal Remote app seems to be adding some smoothing that cant be removed, so it’s not a perfect match the way a Vive tracker can be. That is definitely a more tried and true method with far more documentation on it than this approach (which only a handful of people have ever told me they’ve tried).

hyperbolic films ahh ok I see. Thanks for that. I’ll prolly do a test then get to ordering the vive trackers and go thru those videos on setting up. Thanks!

Yes, you can move in all directions and the virtual camera follows.

It tracks movement and rotation, but a Dutch roll will trigger the iPhone’s orientation sensors at some point.

That’s been answered a few times already, it does both 👌🏼

Hi, I’m not sure how that would be done, I haven’t used realsense.

As far as I know, yes. I think you can earn up to $100,000 based on the license. It says when you download it.

@@hyperbolicfilms awesome thank you!! :) I wasn't sure!

I have not scene a way to do it live entirely on an iPhone. But the app CamtrackAR may be a better way for an iPhone only approach.

@@hyperbolicfilms Well,thank you very much!!