Hey mr Ronald Weasley, I just want you to know that we appreciate your dedication to this channel and we in no way judge you for not making videos as often! You're the reason I got into C++ a year ago and now getting close to releasing my first game!
@@TheCherno Don't know if this will be helpful, but in my experience writing compute code for GPU's (mostly with OpenCL) it is invalid memory access which causes the worst issues (e.g. accessing non-existent array index). AMD cards will often run without reporting issues but Nvidia cards will crash or refuse to even run the code (which is good because it can help prevent damaging the GPU).
I hope you read this cherno. You should realize that your cpp videos have gotten 1000's of people into programming and c++, including me. That means a lot of good softwares made by those people wouldn't have even existed if not for your awesome videos! Your impact is large! Keep doing this we love it
The key is to "give up" in a smarter way, in various occasions I had problems I couldn't solve that I thought I wouldn't be able to and after days of trying and failing I "give up"... but not forever!! What I do is that I revisit the issue months later some times even a year or more later, and to this day I haven't had a single thing I "couldn't" solve. We don't know what we don't know so sometimes just having time go by while learning other stuff may trigger the thing that unlocks you to that old issue. I constantly have 1 or 2 of these with me, in fact just about a month ago I cleared one of those. This proves to me that anyone can do anything given the right conditions, just don't give up those things and instead think about them as challenges yet to conquer.
I take a similar approach when learning gamedev, I'm learning to make games in python using the pygame library and I always tried to implement my own particle system.. but was never able to and I don't know why.. I decided to give up and do other stuff while hardcoding the particles in every prototype (not a single game released but I will participate in my first Ludum Dare in 3 days), after 1/2 month I decided to make my own framework to speed up developement, and when I finally convinced my self that I needed to make the particles.. it workedat the first try.. looking back I did learn some more about classes and maybe that helped, and now I can't even figure out why I couldn't do it before..
That's what I do too. When I'm really getting roadblocked I shelve it for a while and come back later, clear headed and refreshed, and with more energy to go through the hassles of brainstorming and researching. It pays off to give your brain some time to process information, even if subconsciously. It's like working out, really: you have to allow your muscles some rest to maximize your gains. The brain also needs time to get organized. Doing unrelated stuff every day and being well rested helps too.
in my last year of uni working very hard and feeling depressed even though I know I'm on track, I appreciate you making this video, reminds me that even people as skilled as you face the same struggles as everyone else!
I am 100% with you on that one Cherno, sometimes you definitely need to take a break and do something else while your brain quitely comes up with solutions to problems in the background, I can't tell you the number of times ideas for fixes have just popped in my head when I was driving to Uni or taking a shower and I was like "ohh, that could work, I can't wait to try this!!"
Haha... my last crash was solved when I woke up in the middle of the night with the answer. Can stare at code for hours and not see the bug. Go away, clear you mind and it's amazing how often it will be the 1st thing you see.
Potential list of common bugs: a) synchronization - barriers, threading, etc. (check VK barriers and disable multi-threading to see if the issue goes away) b) resource indexing - check if you properly copy the memory of the cube maps. A lot of people make subresource indexing bugs, etc. c) check the constants and uniform buffers you are sending to the GPU to see if you are always sending proper values (use some asserts if some values become unexpected, large, negative, etc) d) check shaders if you have some dynamic indexing going on. If you send in wrong values, you may go out of bounds and access invalid GPU memory. e) check descriptor set management if you are doing something fancy with it, or somehow indexing into it (bindless rendering).
I'm working a a single project since 11 years now. Still fun but sometimes you do great, sometimes you are simply trying to move a wall. Sometimes problems will drain all your energy for a bit. You set a massive goal in front of you. The only thing to really help is a team.
Im am still writing on my game engine and I started right with your series 2 years ago. My engine is not so far as yours because I decided to write my own math library but I know this feeling so well of just wanting to give up! Most of the time when this feeling comes up I take a break for a month or two to get excited again about the project and possibilities you have with your own engine!
Had a tonne of hige bugs I had to deal with in my game Saturn 7 but each time I slept on it and my brain often came up with a solution to try the next day...amazing how your brain never stops...take a break as needed and dont feel too much pressure. Enjoy each little success along the way and before long the thing will be done as expected.
@みやぞのさくら Its great feat that they managed to make such a small game engine, but too many games I've played from itch made in godot run like shit on my laptop. The performance of godot is so abysmally bad.
16:10 There should be a button to match the directional light's direction to the sun, otherwise it's really confusing if you set up the sky for the first time. Looking forward to seeing more of this! :)
I'm sure you'll figure it out, sometimes you gotta take a break or even take a step back and find a new way when getting stuck. but hey you're already so amazing doing all this engine stuff on your own (solo for the longest time) you got it cherno :D
Haven't watched any of your videos in a long time, but I wanna say: you inspired me to get more into C++ and computer graphics and I actually recently finished courses related to both :)
its important to stay motivated! solving those crashes seems not like much fun. it is definitely ok to take breaks. you will eventually fix it :D keep it up!
You've done some great work. I remember last year you were full of a lot of expectations for Hazel and it looks like they're finally becoming reality. You did a great job. Don't underestimate how much work you put in.
I used to have issues with environment maps in my Vulkan renderer. It would be more frequent the larger the environment map was. So a 512x512 cubemap was less likely to crash compared to a 1024x1024 cubemap. It was also more likely on laptops when they weren't in performance mode. I reckoned it had to do with the vulkan device timing out
I've gotten a similar error when working with DirectX 11. If the compute shader is too complex the driver will catch that the shader is taking too long and assume an error happened in the code to make sure the GPU does not get stuck. A way to fix this is to either lower the load put onto each call of the compute shader, or watch for the error to occur and recreate the device so the program does not crash.
@@lazergenix I prefer lowering the load by decreasing the amount of work needed to work with an environment. Especially when it comes to IBL. The environment map for the most part doesn't need to be thaaat detailed for it to be effective. The benefits of working with floating point images
hey Yan, I just wanted to say you are doing a brilliant job. I started watching your C++ vids about 3 months ago, being a c# and java developer your tutorials was exactly what I needed, you inspired me to learn more about Graphics APIs, So much that I started a Vulkan course, I'm not gonna lie, it's hard but I've rendered my first 3D cube and I'm very Proud of it. Thank you for this video, it reminded me that there will be hard times where we feel like giving up but that doesn't mean we should. I wish you all the luck to fix that bug with Vulkan, if I where more experienced I would definitely have loved to help out, maybe one day
7:30 I had a very similar issue with gl when I was writing my test game engine years back, iirc I had a missing semicolon in a headerfile, compiler never complained but leaked somewhere bizarrely with super strange behavior like you are describing.. Took me ages to figure it out, i know your post is old so irrelevant now but for posterity I think I got it with valgrind and a lot of removing code parts one by one.
Few ideas, though I've never worked explicitly on Vulkan issues: - it sounds like some internal state to the API is getting confused - have you tried stepping through the disassembly? - What clues can you find about what memory Vulkan is using, and what its referencing when it crashes? - How does its contents change when it works vs when it doesnt? Can you dump Vulkan's stateful memory in either case and diff them? - Can you provoke the crash in a minimal program? - If not, can you turn off other systems in your engine until the bug doesnt occur? What systems, when present, exacerbate the issue? I'm sure you've tried some or all of this. For inspiration on fixing these kinds of issues I watch the talk "Forensic Debugging" on GDC by Elan R
So in terms of the Vulkan crash, when I'd get the device lost error, this was typically triggered in the destruct as the device is used to clean up resources, but the device was being used still wherever the crash occurred, so the ACTUAL problem is being masked. I'm curious which validation layers do you have enabled? All? Verbose? Also, have you tested this in RenderDoc? it spits out the validation layer errors as well, I've caught a few bugs that way as well.
I know you've only got Nvidia hardware, but it might be helpful to run the open source RADV driver for an AMD card on Linux and see whether it also crashes. There are things like shader trap handlers which may give you an idea of the issue, but even if you can't it's a high quality driver which you can ask for help with. If you can't get it building for Linux you could probably just use Wine. Unfortunately I've lent my desktop with a RX580 to a friend, and only have my Nvidia work laptop with me now. Another thing 1: Have you played around with validation layers? Another thing 2: vkTrace/gfxreconstruct. That way you could send a trace of the commands to people to replay and potentially find the issue without having to download and compile your project themselves.
Cherno, have you tried using NVidia NSight or RenderDoc to debug your GPU crash? I have used those before to debug and root cause DirectX crashes and would imagine they should help with Vulkan as well. If I were to guess, I imagine you might have an issue with data alignment, data packing, or accessing out-of-bounds memory in your Compute Shaders. Since also Vulkan gives you control over synchronization, I would double check your memory barriers as well (a buffer may contain incomplete data). Hope you figure it out. Cheers.
Hi, have to say I totally agree. There are times when I too get fed up with something and I just divert off and add something "shiny" to the software, as you say a self reward. It really does help.
Somehow related, as a totally beginner I wanted to start making my own game engine with DirectX because I didn't like OpenGL that much so I decided to pick Frank Luna's book about DX12 In the beginning I was understanding what was going on but now that I've reached the second half of the book, I'm starting to feel very confused and tired of creating a lot of things, separating them only to implement one little thing. Now I feel you when you dislike low level graphics programming, especially when I see some cool effects that people create in few minutes/hours in Unity or Unreal But I still like DirectX and I still don't want to give up If someone is an DirectX expert and/or has read Frank Luna's books, please write me back
A error that occures randomly sounds like a race condition to me. You should probably check stuff like pipeline barriers and in general resource related things. Since the problem seems to be specific to the environment map code there may be something that is unique to that part of code. Sorry for any spelling mistakes :-/
Have you heard about BGFX? I saw lots of games use it, like Minecraft: Bedrock Edition. It might get out of learning specific APIs but It really makes our lives more enjoyable.
Also there is GFX-HAL, It is good that we have open source low level cross platform rendering APIs, Personally I am currently learning Apple’s Metal API which is a nice API for learning low level rendering but when it comes to making a product I am definitely gonna use one of those cross platform APIs as I can definitely not learn all APIs for each platform and if it is good then why use specific API for each platform if we already have a good HAL (Hardware Abstraction Layer) rendering API
I am a Game Engine programmer myself, but I don't know a lot about rendering (I focus mainly on gameplay programming and tools). It would be cool if you made a video specifically about the various rendering APIs and what are the common points that could be exploted to write an API-indepentend renderer. I know a bit about oldschool OpenGL and some variants like EGL and NitroGL (from my good old PS3 and Nintendo DS days) but I don't know many of the most recent technologies and I could use some videos to wrap my head around those! Thanks and awesome video as always ^^
I can very much relate to giving critical bugs some space and working on something that doesn't make you hate your life for a bit. I hadnt even realised it till you mentioned it; i'm currently writing my dissertation and that experience definitely going in there somewhere.
Maybe I could be of help with your vulkan crash. The debugging ability is a real issue with vulkan unfortunately even with all the validation layers and the driver logs. But after years of writing my own engine and doing very weird stuff using advanced vulkan API calls to the extreme, we end up with a brain that can have a very good understanding of what’s happening under the hood in the driver and what makes it crash. Also, it is possible that your current crash is a driver bug (I have found quite a few in the past two years especially on the Ray Tracing features and I got them to fix it) But if you’re using old Vulkan 1.1 very standard features and you are using latest validation layers, it’s most likely a bug in your code, or in the way your engine runs. When going from OpenGL to Vulkan, an aspect that can bite you easily is memory management and more precisely deleting a buffer or image that is currently in use by a command buffer that has not yet finished, this WILL cause a DEVICE_LOST error. This may seem quite obvious but we never know maybe you didn’t think of it and it’s what’s happening to your engine: This kind of bug is often caused by the fact that you may be running with two frames in flight at the same time so you should delete objects at least two frames AFTER the last frame that used it. Let me know if this helps, or you may contact me and maybe I can try to help. I have never seen the source code of your engine tho.
I also want to program a game engine which can provide an immense cgi quality graphics in conventional system like ps4 xb1 and entry level gpus just like cryengine and rockstar rage engine ithink graphics programming is a emerging environment in software engineering I think its more great and challenging than basic software development through this we can actually understand a true concepts of physics
I really recommend you to integrate Nvidia Aftermath SDK into your engine, because otherwise you're blind to crash reasons. Aftermath is a must for page faults and other hard crash debugging.
I've just wanted to ask for more videos teaching graphics programming. There is OpenGL series, Vulkan and directX should be covered too. There is lack of this sort of content on youtube. We want to better understand graphics and it would be great to learn from You Cherno.
I have to admit I've liked your videos, since I'm making my own engine in C and OpenGL. I hit somewhat of a brick wall about cascading shadow maps, since you mentioned doing a video about that in more detail about how to do such a thing, so I will put my vote in to that if possible. ;) I have a large environment (It's a space combat game on planetary surfaces) so I'm looking for how to do a large area, and yet still have it look good when the player flies close to the surface and would see their own shadow or that of buildings they are going past. I also have dynamic lighting from the weapons and explosions, etc, so I'm not sure it's practical to add shadows to those light sources, but I would be happy with just the sun doing it. In anycase, I know you have your own series of things you need to do, and my problems aren't yours. But you wanted to know what people wanted to see, so I mentioned my vote. Thanks again for doing these videos!
About the Vulkan crash, I don't know if you've tried enabling GPU-Assisted Validation or Synchronization Validation? Those can be enabled with vkconfig or in source code. I've found many bugs by enabling them.
I'm always a bit skeptical of these thicker api wrappers for most anything where it would have any meaningful benefit. The reason being: for where it is similar it probably isn't too complicated, and where it is different the specialized apis are probably much more suited syntactically to the task of expressing what they are, something the generalizing thick wrapper must accomplish in a balancing act between the apis it abstracts. This obfuscates the important bits and trivializes the easy parts even further. I don't mind me some magical code here and there, but there is such a thing as too much magic.
Rule #1.... it's always the changes that will effect nothing that will have subtle side-effects that will take weeks to track down. (As I have found out many times) Managed to get animated, instanced meshes working in mine now with decent lighting. ImGui was a turning point as being able to change stuff on-the-fly and having a nice interface for examining all loaded assets led to a lot of huge performance tweaks. Got loading all my models from like 4-5 seconds in debug mode to almost instant mainly by using a fast base64 decoder on the vertex arrays. I was debating a full binary file format but they can be hard to debug and look at with a text editor. Most of the file is human readable except for blobs of base64 on a single line for the mesh data.
I fully understand that such issues can cost a tremendous amount of time and nerves, but we all know, the next time you will eventually will solve that issue - either by finding the cause or by changing the way how cubemaps are loaded. So keep going and do more creative things, when bugs stops you. I do this in my personal projects all time and most of the time, it helps me getting a different perspective. Speaking of personal projects, i am in the process of writing a multi renderer myself too, which at the moment is at OpenGL 2.1 level and i have no experience with vulkan. So i would be really interested how to abstract the video apis in the simplest way possible. In my current project which simulates fluid particles i have very simple rendering, except the fluid rendering technique which uses SSF (Screen Space Fluid Rendering). But compared to any render technique you have written in hazel, SSF is childs play :D
It's actually kinda cathartic to know that skilled professional devs also get bogged down and frustrated by bugs like that too. And I totally identify with the urge to just work on something else for a while. I think for me it's primarily that I need to work on something that will distract me and give me a quick path back to the feeling of winning, competence, and actually accomplishing something after hours/days/weeks of banging my head against something that I just can't seem to crack. Once I've built back up some confidence and motivation, I can eventually dig back into the frustrating stuff again. I recently had one that had me feeling like I was on the verge of just throwing up my hands and quitting programming altogether. After weeks of frustration I finally took a break from it and worked on new shiny features for a few days. Then when I attacked it again, I finally stumbled across the root cause. It was a gstreamer bug specific to its WASAPI backend that was breaking drag and drop functionality on Windows in Qt and causing crashes because it was randomly deleting OLE objects that didn't belong to it. Not exactly the place I'd go looking for a bug that looked like a UI issue. heh
I've been going about making engines for a few years now, it went from a quad to now an ecs. With that bit of experience I can say yes, game engines are really hard to make.
@@ProGaming-kb9io Graphics & Game dev here. Kor is correct. OpenGL is pretty much dead for games. Apple has deprecated OpenGL 4.1 **years** ago in favor of Metal. While XSeX uses D3D the PS4 doesn't -- nor does it use Vulkan. Vulkan works across Windows, Linux and even Mac (with a shim). I even implemented a sub-set of OpenGL for a few Wii games. While I love OpenGL it has many, many problems. Vulkan is the future if you don't want MS proprietary lockin.
Hey, Cherno can you please give some tips on how to solve hard problems that a beginner or even an experienced developer (such as yourself) can run into while making game engines?
do not quit. I warn you. don't quit. this is a threat from a game dev who believes you have got some major talent even if it is a group project, this is too amazing to die. its almost looking like a unity clone. this is a good thing as unity works very well. fuck ue4 it sucks. I would love to start using you're engine when dx12 has support. please do work on all major versions of direct x 9+ ect as this will give it some major play in the game dev community. I cannot express how glad I am that you have this project going. more work = more likely your gonna get attention from other devs who would love this.
As of someone who tried to make a Vulkan 3d engine in Java, i know how you feel. I gave up since my game was working fine on the one i was already using. But i still intent to change engine when i finally get more knowledge.
Hi dude! Usually when I get those Vulkan crash on my side is usually related to wrong memory access (uninitialized or not mapped texture areas or shaders trying to use UVs in an unitialized area memory)
Yeah just do something else to avoid burnout! Something simple and creative is the best way. I got stuck on a project because I was so fixated on solving a bug without considering taking a break to clear my head. And that led to a terrible burnout situation emotionally and physically which in turn made the project took much longer time than it supposed to. Lol, I earned my lesson the hard! Always take a break and start something entirely new, simple and fun, to help the brain get excited again. Renew your mind as often as you can!
I'd love to know how (or if) you've implemented a scene graph yet. Plus, how you're dealing with transform updates (dirty flags? always updating the hierarchy on changes?).
I haven't done anything with Vulkan compute shaders yet. But I do believe some amount of synchronization is involved. My immediate suspect for anything that seems unpredictable/random would be something to do with that. I've found the Vulkan Community Discord to be quite invaluable if you're looking for help. Somebody brought this video up over there actually.
Hola Cherno, creo que deberías reestructurar el proyecto Hazel: Tu interés son los gráficos por computadora, a decir verdad he visto muy poco de contenido sobre multiplataforma, todo el esfuerzo valdrá mas la pena mientras mas personas usen Hazel, creo que deberías hacer algo similar a Ogre. Que Hazel sea un proyecto para controlar gráficos a bajo nivel, el cual fácilmente puedas clonar y tener gráficos bellos para tu proyecto, y por otro lado utilizar Hazel en un proyecto como Hazel Engine, mantenido en otro repositorio, donde utilizas Hazel como renderizador, y le añades físicas, y todo lo relacionado a un game engine. De ese modo podrías separar lógicamente mejor el renderizador del game engine. Comunmente las personas querrán mas usar tu renderizador en proyectos personales que lidiar con el editor entero (tal vez sea mejor partir el proyecto en dos, es mi propuesta). Claro que el renderizador Hazel debería lidiar con cosas como poder iniciar con GLFW, o SDL, etc. De este modo podrías centrar tus esfuerzos en algo que puede funcionar extremadamente bien (hay pocos renderizadores realmente modernos y bien documentados), y además, podrías continuar con el editor para el que quiera el proyecto y la experiencia Cherno completa :). Se que suena a mucho trabajo, pero mi intención no es animarte a trabajar aun mas, sino a cambiar el enfoque con el mínimo esfuerzo para garantizar llamar aun mas la atención de los programadores alrededor del globo. Y a largo plazo podrías estar trabajando en un renderizador realmente apoyado por la comunidad.
