Create 6DoF Video from 3DoF VR180 using WebXR and Canon EOS VR in the Metaverse

Подписаться 242 тыс.

Просмотров 11 тыс.

50% 1

Learn how to create six degrees of freedom (6DoF) VR video from three degrees of freedom (3DoF) 3D VR180 video with any VR180 camera in the Metaverse using WebXR.
In this in-depth tutorial, we will learn What is 6DoF and Why 6 degrees of freedom is better than 3DoF in VR and XR experience inside a VR headset like Meta Quest 2 or HTC Vive. We will explore new emerging technologies in AI and Computer Vision to transform 3D 180 fisheye videos shot on a VR180 camera like the Canon EOS VR System with Canon RF5.2mm F2.8 L Dual Fisheye lens into a 6DoF immersive experience on any VR platform / mobile browser that supports WebVR and WebXR.
0:00 - VR180 is NOT VR?!
2:38 - Demo 6DoF Video Created from VR180
4:26 - 6DoF vs 3DoF
6:03 - Creating 6DOF is Hard
6:50 - Lifecast AI / Depth Map
7:12 - Fixing VR POV Motion Sickness
7:43 - WebXR Video Player
8:25 - VR180 to 6DOF Tutorial
10:23 - Deploy 6DOF WebXR Video FREE
😎Watch the demo on your Oculus Browser inside Meta Quest 2:
1. Very close up VR180: funky-chicken-222.surge.sh/
2. Medium close up VR180: red-v-raptor-vr180.surge.sh/
3. VR180 still photo: keeleyin-3d-ces.surge.sh/
Download Lifecast VR: bit.ly/3hQUcXR
Download Sublime Text Editor: www.sublimetext.com/download
Download Surge.sh: surge.sh/
Download NPM and Node.js: bit.ly/3pNqOGj
In-depth Tutorial Write-up: lifecastvr.com/tutorial.html
📜 VR180 Masterclass
👉 In-depth Review of EOS VR System: • Canon EOS VR System In...
👉 Review EOS VR in 8K 3D VR180: • [8K VR180] Canon RF5.2...
👉 EOS VR Utility Essential Workflow in 2023: • NEW Canon EOS VR Utili...
👉 Canon R5C vs Canon R5: • Canon R5C vs R5: Which...
👉 RED V-RAPTOR vs Canon EOS VR: / l7ip4zrd3c
👉 EOS VR vs FM DUO vs Z Cam K2Pro vs Insta360 EVO: • BEST VR180 Camera in 2...
👉 How to Film VR180 with Canon EOS VR System (new lesson every month):
Ep1 - Gears: • How to Film VR180 w/ C... (Beginner level)
Ep2 - Camera Settings & Editing: • Film & Edit VR180 on C... (Beginner level)
Ep3 - Edit VR180 w/ DaVinci Resolve 18: • Edit Canon R5C & R5 VR... (Intermediate level)
Ep4 - Master Canon R5C in VR180: • Master Canon R5C in VR... (Beginner level)
Ep5 - Create 6DoF Video from VR180: • Create 6DoF Video from... (Advanced level)
Ep6 - Edit VR180 FREE w/ Mistika: • How to Edit VR180 FREE... (Advanced level)
Ep7 - Get the best Audio w/ Timecode on Canon R5C: • Sound Amazing! BEST Ca... (Beginner level)
Ep8 - Spatial Audio Workflow: • BEST 3D Spatial Audio ... (Beginner level)
Ep9 - Travel VR180 Filmmaking Essentials: • How to Film VR180 / 36... (Beginner level)
Ep10 - 3D Calibration with Mistika VR: • Ultimate Guide: Calibr... (Advanced level)
Ep11 - ND Filter for Dual Fisheye Lens: • ND Filters for VR180 F... (Beginner level)
Ep12 - Computer I use for FASTER VR180 Editing: • BEST Video Editing Mac... (Beginner level)
Ep13 - Build a budget Steadicam System for VR180: • Build an Affordable St... (Advanced level)
Ep14 - Motion Control & Clone VFX in VR180: • VR Video: Clone Yourse... (Advanced level)
Ep15 - Remote Monitor & Livestream in VR180: • Hollyland Mars 4K + Ma... (Advanced level)
🤝 We are open for HIRE! Reach out: hi[at]hughhou.com
Credit:
Forrest Briggs - Ph.D. in machine learning. Worked on 3D VR cameras at Facebook, self-driving cars at Lyft, and robots at Google X.
#6DOF #VR180 #Metaverse

Опубликовано:

23 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 63

@joelface 2 года назад

I think for true 6dof, we need an array of VR180 cameras (at least two cameras spaced apart, but 3-6 seems like a sweet spot). An array would provide the data to properly fill in the holes. It would probably create huge files, but having multiple camera angles seems like the best way to generate true "volumetric video." I see a future where editing video looks a lot like editing video games, where you are manipulating the video within a 3D environment, perhaps even adding in stationary objects like cars and tables with photogrammetry, in the case where the cameras don't properly capture all angles. And if this future comes to pass, where you have about 6'x6' where a viewer can move around a scene and explore, I certainly can't wait to see how the future of film adapts to make storytelling even more immersive and emotionally powerful.

@hughhou 2 года назад

Great feedbacks! I hope the developer can see this.

@johnwate4080 2 года назад

Amazing stuff to see as always - thank you Hugh for posting! I saw btw someone posting a test of 6Dof supposingly recorded on an iphone (don't ask me how) ... but the concept to create essentially memories you can walk around in at some point is pretty cool

@hughhou 2 года назад

Yes it is called Instant NeRF - Nvidia latest research. Check it out. That require crazy GPU power to create BTW.

@nrdythugg4102 2 года назад

Looks very impressive in VR. I even like the glitchy look. Its like a hologramm.

@hughhou 2 года назад

Nice! Thank you for the feedback :)

@justinsuperstar 2 года назад

Found this very very useful.

@sharpvidtube 2 года назад

Still waiting for a good VR180 camera, that I can afford to buy. Surely has to be new ones out soon?

@hughhou 2 года назад

I hope so soon :)

@jongwatcheewakul99 9 месяцев назад

Very cool

@philippkonig9590 2 года назад

which nd filter do you use for the fisheye lens? do you have a recommendation? or do you work without an nd filter?

@rubenlazaro726 2 года назад

Thanks for the tips.

@hughhou 2 года назад

You bet!

@dongyuefu3762 2 года назад

Hi Hugh, I need to purchase a professional 360 camera for work, in specific, taking 360 video of a medical resident training scenario that has two-three groups of people (6-8 people with some equipment) in a 400 sqft classroom. Hope to have high resolution (8k or above), reliable, $5k or lower, easy to use, and less post-processing. I am comparing insta360 pro 2, Kandao Obsidian R, Labpano Pilot EE (or R), or Kandao QooCam 8k Enterprise. Highly appreciate if you can sort these cameras (or if you have other options) by your experience and preference. Thanks again.

@matthewfrew7995 Год назад

As always great stuff Hugh. 6DoF with true 360VR with interactivity is the Holy Grail. Synthetic media is probably the way to go. As an academic working in transformational DARQ technologies and accelerating Metaverse synthetic media and whole synthetic humans (all underpinned with powerful AI) is how 6DoF experiential ecosystems will bring future metaverse to next level

@hughhou Год назад

I would love to learn more your research. Feel free to reach out to me on Instagram @hugh.hou

@BenEncounters 2 года назад

Would make more sense to block the view at neck level though, so you simply get the benefits of comfort mostly without loosing the original point of view and avoiding those terrible occlusions and distorsions. Just for 180 premium comfort basically.

@hughhou 2 года назад

Great idea!

@AgniesMTDronesandPixels 2 года назад

Hi, thanks for awesome content - would you be able to do this with a 360VR footage(like from Insta360 Pro)? Also thinking about 6DOF - I saw your other video with 2 Obsidian cameras put together to create 6DOF. Do you think it would be possible to sync 2 Vuze 3D 360 cameras (one on top of the other) to create the same effect (obviously cheaper than 2x Obsidian...)? Would it be worth looking at?

@hughhou 2 года назад

Not yet. The developer is working on it tho.

@PP3D 2 года назад

Thanks Hugh 👍

@hughhou 2 года назад

Any time

@HealingG1 2 года назад

Thank you~😁

@hughhou 2 года назад

You're welcome 😊

@Kumarswamy_Hosmath 2 года назад

Ignore motion (of subject), what if we capture multiple still images (180/360) - can we generate better 6dof experience?

@hughhou 2 года назад

More camera in different angel, better experience for sure.

@JoleneSugarbaker 2 года назад

With HumanEyes out of Business (Vuze) : They claimed they have 70 Patents on 180 VR 3D functions : Do you think this will hinder future cameras? Their site is still strangely set up like they are still around.. Wonder how wide these patents are... and where these go now..

@hughhou 2 года назад

Mmmmm that I don't know.

@RichardTallent 2 года назад

Interesting approach. With better AI handling the "fill," it could be a good way to remain basically stationary but improve the immersion. Hopefully they'll at least improve it so background information revealed in one frame to fill voids in another. Hopefully technologies like Meshroom will continue to improve so we can get a true stereo volumetric experience from just a few still photos (much further apart). I'm also looking forward to seeing AI's that can make reasonable "guesses" about symmetric objects, continuous forms, and computational textures so we don't need to capture all sides.

@hughhou 2 года назад

Great feedback Richard! One thing I have not mentioned and it is the ability to extract 3D module or point cloud from VR180 video into Unreal Engine using this approach. So far only 3D 180 photo but video is coming next so I did not mentioned this yet. But this is close to your direction - we can potentially extra data and use other software in improve the data and "fill" more holes with machine learning. When video feature is out, I will combine it with an unreal experts to further dive into this topic. I plan to go deep, very deep on this topic for sure.

@WoutStandaert 2 года назад

I am less optimistic for a photogrammetry based solution to 6DOF video. The textures won't be PBR, so that's a huge step that's missing already, and it doesn't do translucency, refraction, reflections and volumetrics. I think Google is on the right path with their DeepViewVideo AI project, seeing as their "welcome to light fields" demo are still the best 6DOF photo's I've ever seen.

@hughhou 2 года назад

Oh yes light field is great! But so far it is impossible for video after Google took Lytro apart :(

@Octamed 2 года назад

Looks exactly like the Minority Report 2d to 3d holographic projectors. I knew at the time that'd it'd look exactly like that. Although I've recently seen some totally insane AI fill algorithms that would easily fill in those gaps.

@nebulaxu9838 2 года назад

Hope RU-vid support this sooner.

@hughhou 2 года назад

I hope so too!

@zebius4157 Месяц назад

Besides price, what would you say the difference is between lifecast and kartavr?

@hughhou Месяц назад

They are different software and do totally different things.

@zebius4157 Месяц назад

@@hughhou I ask as one is able to derive 6dof video from kartavr, is it delivery for webxr?

@jakemathew 2 года назад

The glitchy holes that appear behind the subject as your move your head is distracting. I briefly chatted with the developers of Lifecast and their software supports adding a clean plate background layer. So you would first shoot the environment without the actors and any foreground elements. then you shoot footage with the actor. Then the software will fill the holes with the clean plate. Hopefully, future updates of this software will easily allow users to upload clean plates.

@hughhou 2 года назад

Yes exactly! The version of the software I have when doing the review does not have this feature - but the clean plate can really filled those holes.

@jeromelozano 2 года назад

Our 6DOF music video created in collaboration with Lifecast : lifecastvr.com/mexhhh.html

@domonicwhite 2 года назад

Exciting but doesn’t look fit for purpose just yet.

@hughhou 2 года назад

Indeed :)

@michaelsiebielec5554 2 года назад

Great concept but the artifacts totally ruins the experience.

@hughhou 2 года назад

Yes it is still very beta stage. We will need more learning, more camera maybe to capture more data to fill those holes :)

@misterhorse8327 2 года назад

The artifacts are imperfect, but feels SCI-FI/Cyberpunk-ish.

@hughhou 2 года назад

Yes. Def lots of room to improve. It is a step into the right direction :)

@fidobarks1164 2 года назад

I really appreciate all your content... BUT... You gotta stop saying metaverse... you say it all the time... you are smart enough to know there is no metaverse yet. A 3d video is not the metaverse. Neither is a 360 video. Just because you see it in a VR headset it's not the metaverse. The actual "Metaverse" doesn't exist yet. Maybe one day, these videos will exist in the Metaverse but they don't right now. They are on the internet. Stop using marketing speak encouraged by Facebook. No one is even going to call the metaverse the metaverse because Facebook is so stupid. I do appreciate the content though, really!

@misterhorse8327 2 года назад

Nah, the metaverse has a nice ring to it.

@fidobarks1164 2 года назад

@@misterhorse8327 It's just the recent insane frequency and total misuse of the word in his videos that drives me nuts. There are legit shared virtual spaces that are "trying" to become some aspect of the metaverse... but man, stuff you see in a headset isn't the metaverse, it's VR.

@hughhou 2 года назад

I should tune down using that M-word. I am using it just for click-bait purposes and also my Oculus TV channel is called Metaverse Stage.

@misterhorse8327 2 года назад

@@fidobarks1164 But it will eventually be applicable since the "metaverse" encompasses VR, AR, MR. His videos will still be metaverse worthy since in the future people can watch his videos on Apple's AR glasses or Magic Leap's AR glasses.

@fidobarks1164 2 года назад

@@hughhou Yeah I get why you are putting it out there - just maybe sprinkle it in a little less in the content. Honestly I'm pretty sure Facebook ruined the idea of anyone EVER letting the word Metaverse become a real thing now. We'll end up calling it something elses no multinational corporation will want to trademark... Boobietown / Dicksville / Poopland or somehtng.