A cautionary tale about ChatGPT for advanced developers 

I have a newsletter, you can join it if you like: mobirony.ck.page/4a331b9076

Copilot keeps randomly "forgetting" that it's integrated in VSCode and has access to my files and I can't convince it otherwise only start a new chat.

As a solo dev, this is how i use it as well. Asking for hints and general pointers as well as code review, but not trusting it to write code i use directly.

What you describe here is more broadly known as automation bias. People tend to overly rely on the outcomes of automation, even if they have instruments at hand to know better. The fact that you critically review the output of the AI and understand that you need certain skills to assess its output goes a long way with that. I really like your example of knowledge gaps and I'm stealing that for my lectures.

Makes me think of my current job. I work for the government, and management keeps pushing us to be a "directing organization", meaning that all the knowledge is at external contractors/suppliers and that we steer them and check their work. The idea is that we have broad, general knowledge, and specific domain knowledge is at the suppliers. The issue is though that many suppliers are unreliable and produce shoddy work, and we have no way of recognizing that unless we have that specific knowledge ourselves; which can only really be gained by doing the work that the supplier does (i.e. actual development). So how are we as humans to check, correct, and steer AI when it starts doing all our work for us?

I rarely use chat GPT at all, but surprisingly today I've used it first time since a few months. I was stuck trying to finding a solution to my problem and tunnelvisioned one scenario too hard, cause I thought I was close. Resigned, I asked gpt what it would do, described scenario as best as I could and it actually returned me a very useful answer. It was similar to my previous train of thought, but it filled the gaps, similarly to what you've showed in the video. I think that gpt might be very useful in those kind of situations, where you know that you're close, but you're missing something or need other perspective. Thanks for the video!

As a developer for more than 40 years, I have experienced more frustration with AI than solid solutions. This video does a great job of summarizing my own current approach to using AI at this stage, and serves to provide confirmation that it’s “not just me” having the issues. Thank you!!

The problem with current gen AI is not that it kinda sucks, but that it doesn't suck the way I like to be sucked.

There is a huge black hole of bad for many people using the "AI" to help them. I'm currently tracking more then 2K open source projects that has more or less the same code snippets in their repos that contains easily exploitable code.. I'm expecting 2024 to be the year of massive issues with systems running this code on production. It's like people unlearned basic "safity" coding habits. Great video, think it is important that people who are public with their coding skills do this. Good on/by you! 🙂

You hit the nail on the head with this - I've had the exact same experience. When my knowledge gap is too large to know any better, I actually tend to waste time using LLMs rather than save it. Also, my experience with Copilot was so frustrating that I had to turn it off within a week. It was forever making terrible suggestions. Perhaps that's because I was using a lesser-known language, F#, but ironically I understand large portions of Copilot are written in F#.

What is the deal with typescript and why is the code so exhausting to look at?

My job grants us access to Copilot Enterprise, which allows us to “index” our monorepos such that our instance of GPT/Copilot has the full context of our specific codebase. It *helps* over blind questions to ChatGPT, but it still has a tendency to recommend more generic/unhelpful solutions which just match our typical syntax/nomenclature. It’ll do a solid job of writing PR descriptions too, but only handle the “what” and not the “why”

CoPilot has saved me a lot of typing of C# code in Unity. When having to add a lot of names to textures it knows what your doing and predicts the next name for example.

Issue with using gpt is you can start a project with it but as it grows you realise that the foundation is just wrong. Then you have tk start over

Sometimes I think I'm helping ChatGPT, not the other way around. Regularly it provides not only incomplete, but plainly wrong solutions, even without knowing the full context. AI assistants might become the next weapon in war by providing unhelpful suggestions to the ones depending on it due to a lack of knowledge...

What I do is write my code, then ask chatgpt to critic my code, then ask it to provide an improved version using the latest good practices, then I redo my code following what make sense. When I can't manage to make something that work, I copy a similar code that work and ask chatgpt to explain to me the parts of the code that I dont understand (if the code is in javascript and short, chatgpt explanations are almost always correct), then I do it my way, rince and repeat. I also use phind, which is very good too AND provides sources (for example stackoverflow) to its explanations. IMO a good trick is to provide your code to chatgpt, ask him to describe what you are trying to do, then use the output of chatgpt as a prompt for other LLM to get various ways to solve a specific problem.

I used ChatGPT to help fill knowledge gaps for a C++/TS WASM module and i ran into everything you stated. If my knowledge of C++ and TS weren't up to par i wouldn't have been able to see how the solution provided was generic and not suiting my entire needs. The solutions look like they work but they often need you to meet halfway in the middlle to pull out the concept.

Great vid. I think it highlights that ChatGPT can’t create something out of thin air to solve your problem, but draws from a pre-built deterministic model. It’s not a CSP solver yet. If it hasn’t encountered a solution in real life, or the uncommon solution is outweighed by more common solutions, you won’t get what you need without a lot of prompting. The other thing to remember is that often it is just as fast going direct to documentation.

💯% agree - using it to fill small gaps. Delivering stuff is much faster and stronger (ui stuff, unit test cases after corrections) but when tried to use ngrx where I have little experience - it failed one after another. After learn something more on that topic Gpt is now helpful, but my role is to keeping direction, not to try what he 'thinks' is good blindly. Good job Joshua❤

Beautifully said. Breaking things down for AI usually helps me when I'm stuck, but I have also implemented a hard 10-minute rule of chatting with the bot for an answer. If it's clear that it also doesn't know what I don't know, I have often found the answer with a simple google search in minutes after wasting hours trying to use AI as a "know-it-all coworker".

It took me an embarrassingly long time to start using ChatGPT. I only started using it a couple weeks ago. It's been amazingly helpful. I absolutely agree, the more complex a question you're asking, the less correct the answer will be. I use it mainly as a way to remember certain concepts that have slipped my mind, syntax, class names, implementations, etc. I don't think I've ever asked it to make some specific code for me, as sometimes I ask it what I think to be a simple question, I can get a complex (and often wrong) answer. I dread to think what it would do if I asked it to do something I couldn't verify myself.

It's also important to check the generated code, even on simple examples. I was recently calculating the accuracy, precision and recall for a categorization problem, literally just (part amount)/(bigger part amount) to get percentages. PyCharm's integrated AI had suggested the formula for accuracy correctly, so when it suggested one for precision i assumed it was correct as well, and showed the results in our weekly. Only to realize that i had the formulas for precision and recall mixed up, turning my results from 60% effective to 13% effective. Quite different from what i had tried to present.

This is what I've come across with gpt solutions to coding issues. I found that if it's something small (like building a function that loops through predictors and plots them with a target) it will work quite well. It has even helped me dive into some coding concepts I would not have done expediently, such as learning RShiny. But when it's the complex stuff where you need more knowledge, chat gpt fails. With me learning RShiny, the gist of it was pretty clear and easy, but then I wanted to make it so when people clicked on the map, it would render and display a markdown document of the local figures that I generated for each location. Chat GPT was completely incapable of providing the solution. I had to learn about parametric rmarkdown and observe event complexity in order to get a solution, and chat gpt failed to recognize my goals and instead wanted to brute force things in ugly, inefficient ways. You outline it perfectly. It's great for small gaps. But it's really messy and bad for large gaps.

I always feel that you can't rely on LLMs for things you don't already know decently well. As you say, it is fine if there are small knowledge gaps. I use it in different manners: · Small knowledge gaps - clarification, examples, insights on my code · Large knowledge gaps - starting points for studying, broad information to help me navigate a new topic · Silicon duck - bouncing ideas · Structure checking - does the LLM understand my code intent? · Brainstorming - LLMs are great at detracting from the focus which is at least useful for brainstorming sessions. Thank you for your insights!

That idea of smoothing out the small gaps is spot on. I get stuck far less now than I did a year ago. Also just talking about a problem your facing can lead you to a solution even if the AI can't come up with it.

I am not a programmer at all. But I decided to start experimenting because I know I have chatgpt to help me along the way, otherwise I wasn't even start. For sure I will suffer way more compared to someone that has experience in programming. Everything I tried has worked so far, so I am learning and doing things that are more challenging over time. My use case is automations for my job. I love the challenge of break a big problem in small parts and the satisfaction received after solving each part

I tried it. I asked it a very straightforward question: "what function can I use in mongoose to update multiple records and return the result in a single step". It's not possible, MongoDB doesn't work that way. Anyway it just imagined something that didn't work with a high level of confidence, despite the API being well documented. If a junior did that to me multiple times, I'd fire them.

Excellent video, exactly points out what I experience the whole time while developing^^ It helps me personally to often refresh conversations, using the classic GPT4 model, using API and providing the model as much context about the coding issue as possible

Amen, Brother. I always know the second ChatGPt starts suggesting "Drastic and Sweeping Changes", it doesn't know what it's doing. That's where decades of programming experience, along with a clear knowledge of what I am trying to achieve and how the entire program is structured so far, helps me come up with a more appropriate solution. And here I thought I was the only one going through this!

I've been using copilot for over a year and have yet to fall for the pitfalls you suggested might happen. I have copilot configured as an autocompletion source (next to LSP-based auggestions). I find myself reaching for it mostly when writing boilerplate stuff that I know how to do, or when I don't know the API and would like a starting point that i can then look up in the documentation or original source. I guess it might be different for other people, but as long as you don't start blindly accepting copilot suggestions it can provide a valuable speedup without too many apparent downsides, IMO.

After creating an algorithm for a geometry problem I asked ChatGPT to solve the same problem, it generated nonsense, I changed the problem explanation to try to be clearer and ChatGPT changed the problem to generate nonsense again. I was not even asking for code, just how you would solve the problem.

This video is a testament to your proficiency as a youtuber and I appreciate you sharing it.

Knowing context is very important. I often shift ChatGPT in the right direction by saying things like, "Allright, but what about this approach..." This way, it also implements it in the way I thought of and compares it.

Your comics is great illustration for the overall AI coding assistance experience this is the exact same way I treated AI at this moments but as a good practice I always provide ALL the information needed and my worries and my expectation to chatgpt before i ask him

They are great tools to speed up processes, in particular simple tasks like documentation, small functions with limited scope and style suggestions. Otherwise you have to be on your game!

This perfectly outlines the problem I often have with chatgpt. When using it to solve problems which don’t have a defacto solution and require lots of context, you can easily fall into rabbit holes where further prompting gets you further away from a realistic solution.

I started using chat gpt exactly in the places of this thin lack-of-knowledge area, but wasn't aware about it until saw your video - a great summary of how I use it now as a tool in solving a practical narrow task. Cheers!

I completely agree. As a university student learning programming, I find having ChatGPT by my side to be a great tutor. It’s a better experience then regular search, especially since I like to ask follow up questions, examples, and some new ideas it can think of for me to explore. I find it’s also great for boiler plate and I can focus on the more important aspects of my code. However, I try to make sure I am not relying on it so much where I am just copying and pasting.

I just want to point out that I have recently used ChatGPT in the way you suggest not to. You’re absolutely correct. I was able to brute force my way to a solution, but I was only able to solve the more minor issues after I understood how the code it suggested works in the first place. It’s not a “bad” way to work, but it is frustrating. At the same time, I’m not sure I would’ve gotten to this point without it.

The gap analogy is perfect!! Thank you for this insightful vid.

Hands down agree. The only thing I would add is that it is possible to get ChatGPT to do much more/better than it's initial response (and even without human feedback using agents). However, I find that I often have to run things back through it so many times for both error and efficiency checking that by the time it gets to where I really want it, I've spent more and taken longer than made sense. So I've defaulted to using it just how you illustrated except when I'm just wanting to play around (and have the time to do it).

100% wholeheartedly agree. I just used it to learn k8s, but I didn't just let it write the yaml and called it a day. Instead, I asked it the questions that it would've taken days of google search to condense by myself from the plethora of resources out there. At the end of my learning journey I could fix its mistake and even propose it improvements over what it had given me at the beginning.

Lot's of devs I know are actully turning off github copilot in their IDE, sometimes the results it gives us will be somewhat logically wrong, but we won't see it, and the Tab key is also too adicting to press I'll consider turning it off

This is a very good video on how and when to use these AI coding assistants. It had to come from someone with experience with the tools to be able to add that kind of perspective. The cautionary notes on avoiding the "traps" is very much appreciated. 🤔

100% on board with this assessment. It's an assistant, a term I hadn't thought of even though I'm driving the code. It can get a pseudo understanding of what you want to do, but unless you explain in detail what you want to do, it's going to decipher your input and then send you output then... that's it. It doesn't ask questions or clarifications. It just assumes it knows exactly what you want, everything you gave it is the golden truth with no other outside bits of information that are relevant. It's a tool. A very ... inaccurate tool at times... that jumps to a lot of conclusions.

How are all these people using chatgpt so seamlessly in their jobs. Every time i tried myself, i end up answering my own question as i am typing it or if end up actually asking it returns a mediocre piece of code that may not even work until i debug it. Make good boilerplate thought but there are much simpler tools to make boilerplate code.

I’m too stubborn to even consider using something like copilot, but I did have a couple of chats with gpt. Based on my limited experience, it’s usually okay to bounce around ideas with, but it does feel too much like a “yes man”. And it can’t even insult you for your stupid ideas, which I find annoying. It easily solves common coding problems, which is great to have in one place, but is only really useful to get going. That said, in general I had this feeling of nothingness after chatting with it. Idk if anyone else gets that.

SW architect/full stack dev here. I greatly appreciate your insights, as they perfectly articulate my stance on the issue. I've encountered the same phenomenon: with the help of GPT-4, I can create something that somewhat works in a language I'm completely unfamiliar with. However, I find myself at a loss when it comes to resolving any non-trivial bugs. On the flip side, GPT-4 has significantly increased my efficiency in environments I'm already proficient in. My concern lies in the implications for aspiring developers. With senior developers finding less need for junior support, how can newcomers to the field gain practical experience? This raises a critical question about the pathway for students beginning their careers in development.

This is exactly how I use it as well. I'm a student myself, so my focus is on learning through the tasks I'm given. Which is why I tend to only use ChatGPT if I don't understand an issue I'm seeing, or if it could give me a summarized explanation of some specific topic for which documentation is kind of fragmented. And Copilot merely to accelerate my typing, in only having it finish lines of code when it suggests a long line/block of code I intended to type already in the first place.

Quite good helper in testing :) When I'm sure that code works (not complicated), I just ask LLM to generate tests and then check line-by-line what is generated. Saves a lot of time. Also, sometimes routine tasks, like "replace this If () else { if ... } with "switch..case" - does it excellent :)

Exactly. OpenAI API key plugged into an extension/own front end shell for occasional brainstorming is the way to go. While the copilot suggestions for repetitive code autocompletion is nice, I find it to be buggy far too often.

Great video man, it's a fantastic sidekick, but from someone who has used it to kick start my self-taught programmer mission, it definitely doesn't "do it all for you" for but it most definitely speeds up the process.

i use it alot more for helping me to LEARN than to actually code FOR ME. I use it to understand how to STRUCTURE things. It has taught me alot. It's kinda like having a coach, that you can askactual questions to and get actual answers. it's not 'intelligent' tho. at times, it will just widly, randomly, get shit REALLY wrong.

Great take, I agree completely. Also great to catch a fellow South Aussie in the algorithm :) I'm loving GPT-4-Turbo via API (Chatbox as a UI), using it daily in rubber ducky mode. The biggest gap for our needs is efficient context summarization IMO, but that can take us mere seconds of copy-pasting relevant blocks

That's a very good point you give, I've used it myself and found that filling in the small gaps has been far more helpful then asking for a complete solution that I have no understanding how it functions. Especially when it fails and GPT can't reliably guide you to fix the problem that it created.

I love the visual here, that sums up perfectly the usage behavior and its potential issues. Well presented.

If the answer isn't the direction you hoped for, it's often a sign, that you didn't provide enough information, restrictions or requirements. It forces you to really think systematically and to analyze your own understanding of the problem in a structured way.

I comp this 100 %. I'm not a coder, but I'm an electrical engineering student and i've noticed that these LLM models really shine when you have appropriate knowledge, context and understanding to ask them the right questions as well as know when one is obviously wrong with the answer. They add a speedboost to information gathering (instead of having to manually google every single thing) and also function as a substitute teacher for specifics of topics that you're curious about. Overreliance on them as you would rely on a microwave to cook your meals is a catastrophe waiting to happen and I think it's worrying what kind of market hype is forming around them right now.

ChatGPT single handedly taught me compute and fragment shaders in js and c (seperatly, not at the same time). When you have literally no knoledge where there are many tutorials out there, ChatGPT is amazing when it comes to consisely regurgitating only the good bits of all the tutorials

Can chatgpt help you pronounce consume?

I recently started using it just as an assistant and it’s pretty good! I’m writing in a new language so whenever I have the idea of what to do (but not the exact syntax), being able to instantly get the syntax and what I mean across into code is just really helpful compared to Google searches vaguely trying to get my point across and scouring forums. I think I’d have found the info either way, but it just saved time for me.

this is my same exact experience, it definitely gets up to very fast speed if you have tiny gaps in knowledge. Originally coming from flow i wanted to achieve similar generic conditionals in typescript same as you, it split everything up to make it easier to understand but ultimately it never gave me a solution that worked but gave me enough to experiment. And after reading typescripts docs on generics and inference hundreds of times i actually think there’s not *enough* documentation about specific nuances (like type broadening from literals to string type causing inference errors). In the end what worked was very small and simple even when chatgpt asserted that i had reached the very limits of the type system.

I've had similar experiences in the field of machine vision. For example one time the answer I was given did get the job done. However the premises didn't allow for too much optimization which was became a requirement later on. I had to go back and rethink the entire approach in order to reach the required runtime. As a result, these days I use ChatGPT in two ways: 1) I ask it to do the leg work on something that I can fully describe (i.e. write a function that takes in XYZ and by using ABC return 123); and 2) to list known methods for a problem and to sort them by accuracy, runtime and license. This is a real time saver and I've been able to avoid the traps, at least until now :D

I'm by no means a professional. I only just started coding a few years ago. GPT is great, but not only can it lead humans astray, humans can lead it astray. It gave me a straight-forward answer to a problem I had been having, but I was pretty sure I had already tried what it was telling me, so I told it that it was wrong, and why, and I explained my problem in more detail. It agreed with me, told me how smart I was, and integrated my own words back into it's solution. Before I could entirely implement the solution, however, I found out that GPT was right the first time. GPT cannot replace substack for that reason. If that interaction had happened on substack, NOBODY would have been telling me how smart I was. And that's a good thing.

I’ve been trying to use copilot for the first this month and I have not really let it fully take the wheel, just basically using it as a smarter autofill to help reduce myself having to type out full objects, names, etc. I think this is where it’s useful and should be used for, rather than writing entire functions and relying on it to complete the core functionality of your application. Don’t let it build the entire car but let it help you screw in those bolts after you laid out the parts 😊

you articulated perfect what I have been doing since GPT 4 has come out at my job. i use it as a quicker way to google stuff. it works amazingly when i give precice instructions and context, but if i ask it somthing vague or about somthing I am not well versed in, I see a sea of red errors. i also use it to help figure out what an error message could mean, it seems to be very good with that or at least breaking it down and then a lightbulb moment happens and realise what is going on.

I'm working on an indie game rn, and have only been at this for the better part of a month, but this is exactly how I've been using chatgpt to supplement my knowledge. I never ask it to do something I don't already have a decent idea of how to do, and I never use its code directly, always typing it out fully myself and making sure I understand every line. It's been really helpful as a steering wheel, but it's my foot on the gas at the end of the day.

As a person who didn't write a single line of code, chatgpt has enabled me in understanding how code works and helps me communicate better with developers. I'm able to convey project requirements easily.

I've actually had GPT4 itself tell me, that I should not expect it to completely write whole programs for me from start to finish. Claude taught me what I needed to know about using stacks in Python, and from there I've gradually been developing my own stack based style. Claude gave me some hints, but the way I incorporated them has been my own. Claude actually has a very strong, implicit OOP bias, and was telling me that I would benefit from OOP. Claude will, however, still give me information about how to do things functionally or imperatively if I ask.

One tip i have about chat gpt is to end every query with "be concise". Gives you much shorter answers with less useless fluff. If you need a long explanation then, as you said, the knowledge gap is usually to large.

This is all well thought out and very well put, thanks for sharing this! I've been approaching the tools in much the same way, by trying to help bridge knowledge gaps, but then trying to backfill those gaps once I get closer to my goal. Sometimes it's hard to simply have the words to describe the issues I'm running into, and having that virtual conversation partner is a great way to get unstuck.

This is how I've always used it intuitively. I am in school right now, and whether working on assignments or personal projects, I would only refer to it for either hints (I tell it specifically not to give me the code), or just ask it to explain a concept to me. The idea of getting it to do all my work for me scares me in a way because I know I will come out the other end of it with phony knowledge and work.

As a junior dev trying to play catch up, its extremely helpful having a more powerful Google to answer most of my questions. It's also super fantastic at making lots of changes that would take me much longer by hand. For instance converting vue2 to vue3 syntax, although you still need to check everything it gives you in the end.

This is why its useful to tell ChatGPT a general idea of how to solve the problem more broadly

ChatGPT will create an immense amount of work for developers who know what they're doing, because people who don't will start building critical systems with no sense of the danger. A good time to be a consultant.

Great video: presents a very real problem that hsa the potential of being overlooked, and a concrete example along with it.

well thought out and presented. subscribed.

I use ChatGPT in my game development quite a lot. I agree with pretty much everything you say here. On the current project I'm working on, (a massive GPU accelerated particle system meant to operate as a cellular automata simulation) ChatGPT was invaluable in bridging the knowledge gap needed to begin writing effective shader code. It's also been extremely helpful with 3D math, which is a huge part of the sim, and something that doesn't come naturally to me. That said, I've definitely seen GPT's "just scrap everything and do this instead" attitude more than once.

I think this is totally accurate. I am definatly not knowledgeable at coding and i got some amazing results at different points tryingnout CHAT GPT at different stages of its development but those results were far and few and very inconsistant. There were some times where i knew enough to know that parts of the code sounded good but didn't make sense. And no matter how discriptive i was the suggestive code that was given to me was always plagued with errors. Now that i have waisted time i have to learn proper scripting concepts before trying to move forward with some projects.

Even as a beginner coder it didn't take me too long to realize that I could write more efficient code than it could. But like you said, great at helping solve specific problems or leading you to a solution.

Thank you for this. Finally an analysis that doubles with my own experience using chatgpt for coding and not one of the usual wide-eyed "it's better than 90% of programmers" pieces. I've tried using gpt 4 a few times for my own coding problems and came to the conclusion that in the time it takes me to give chat for enough context and debug it's code I could easily implement the code myself. And on problems where I was really stuck it did exactly what you described and suggested a seemingly plausible approach which then didn't work. Once it even made up C# methods that don't exist and assured me they would work. So yeah, it's good for basic stuff, especially if you are working with a new framework or language, but for anything more complicated it's like asking a lying coworker who doesn't know what you're working on for advice via teams without being able using screen sharing.

I agree completely! Im first year masters AI student and have BSc in ai where the focus was on computer science. At the start of my masters I promised myself to not integrate AI tools in my IDE. Now i mostly use it as an information retrieval tool to give suggestions for simple functions or to help debugging. Some of my peers with a different background deliver code that is a patchwork of CGPT outputs and are only able to explain the functionality of their code at a high level, i think many beginning coders will fall into the CGPT trap

I always reformat working solutions from Chat GPT into my own coding style and syntax, so I can ensure whatever code GPT generated works and stays consistent. Never just dump raw code from GPT and call it a day even if it works. Because you’ll come back in a few months and have no idea why or how that piece of code works.

I used ChatGPT/Bard to have it produce SIMD variation of my code. There is a sea of SSE instructions to comb through that LLM quickly showed me what I wanted. On the other hand I asked if it could multithread a bit of code. It almost worked, but in the end I just had abandon LLM and read the manual + stack overflow.

I experience what you describe here regularly and your lesson is extremely important for people to absorb. My rule is that when I feel like I'm trying to squeeze water from a stone with Chat-GPT I take a step back and re-evaluate what I'm trying to do. This is even more important when it's not concerning knowledge gaps, but a complete absence of knowledge if you're trying to do something you've never done before. We know that Chat-GPT can be extremely helpful when you know the specific problem you want to solve. But, when you yourself are clueless Chat-GPT will definitely mislead you and waste your time. My tip for everyone is to research first, try second, and then Chat-GPT when you are at least not entirely clueless.

ChatGPT is trully an assistant, i find myself yelling at it, and telling him how to do things, but sometimes it really does the job itself and surprises me. For experienced developers, it can be frustrating to just issue orders when you kinda know the outcome and it does the oposite. I think we are expecting too much of it, it's very good already, but, it's still has a long road ahead.

In my experience, ChatGPT can lead me down a rabbit hole sometimes where I wouldn't go otherwise but I learn a lot along the way. Even if my project ends up imploding, I understand a lot more by the end of it than when I started. I think the biggest power ChatGPT has is as a learning tool.

I have found that an additional use for ChatGPT is handling repetitive, arduous tasks. For example I would provide a list of things, do whatever needs to be done for the first and have it prepare the rest for me. That has saved me a lot of time, specially when refactoring old code.

Very real. I've seen people with large gaps use it but as they lack understanding it's basically blind trust in GPT, which doesn't end well. Most of the time good usage of GPT is either idea generation (getting unstuck) or simple stuff like refactoring simple code. Anything harder and it's pretty pointless.

I use co pilot to do small repetitive tasks, I have not used it to its full capability after 3 months of using it. I use ChatGPT like you said, for those small knowledge gaps and thats a fantastic way to put it. Very well said. It important to have a knowledge base because the code provided by ChatGPT, or even copilot, is not 'ready' to use but after a little tweaking it provides great starting points. Another thing is these AIs can only take small segments at a time and feeding them multiple scripts to modify at once hasn't proven itself useful yet. You still have to push the weight as the developer

@JoshuaMorony - you are one of the smartest developers I know 🙏 I am honestly so glad to hear you saying that, because my interactions and workflow with ChatGPT have been so similar. Probably not the same level of code but just things go exactly the same way... I have self doubt but eventually I just go with my own solution especially in the cases when it suggests pretty much re-writing the whole document just because this is the more "general" approach... But it does help with the little gaps and mostly - when ChatGPT suggest something, then this helps me and inspires me to find another 3rd, 4rth and maybe even 5th approach :)

I get what you are saying but your example might scare some people due to its complexity, so I will tell you my bummer experience with it. I have relying on 20+ year old code to handle dates with gregorian years, leap years, daylight savings and stuff, and decided it was time to update to a more up-to-date (it doesn't help the code was in old php 4.0). I knew how to do it, I had experience and could very well just sit down and do it, but you know ... time. So I thought hey, this should be simple and GPT can probably do it very easy. So I asked it to create functions for each specific date/time conversion needs and for a while, I was pretty satisfied with the answers, I understood them fully, so I didn't see why not use them. By the time I had 4 different functions almost totally written by GPT, things went haywire. Things started breaking and I couldn't figure why. So I started to debug each function and sure enough, there were a TON os exclusions, edge cases and non trivial events for which the function wouldn't work. For instance, I asked for a VERY BASIC FUNCTION to tell me how many days there were between two dates, easy right? I looked at the code and seemed ok ... except I missed that it was ignoring the YEAR, the year was in there, but the math would essentialy do something like Year - Year, so it was giving me the difference of two dates ignoring the year. Very stupid mistake. As I started showing this to Chat GPT and asking for a fix, it started bloating the function with very dumb over-the-top checks that were totally not needed: instead of using the years on the original math, it started creating conditions for when the years were different, higher or lower. And it kept getting worse as I found issues with leap years, even leap days (this year we had one), so I just tossed the whole thing in the garbage and did my own ... in 2 likes of code =x Moral of the story, it will do exactly what you asked, and since you didn't specify a huge list of conditions to look for, it will give you something that works only for the case you asked. It can't figure you want the full solution. All functions it made were wrong and I had to write myself, I ended up losing more time finding the bugs, scrapping the whole thing and writting my own (always faster and smaller than the ones proposed by GPT). Now I only use GPT to give me suggestions or as a substitute to reference about functions, but not to write code.

You really put it into words. Been the same journey for me

Yea I had a similar experience with this when I asked ChatGPT how it would approach solving what had been my capstone project at the time. To my surprise, it gave an incredibly detailed plan which mirrored my own teams plans quite closely. By this point we had already developed most of the project so this was merely to satisfy my own curiosity, but I wish I had done this sooner as it could have saved us weeks of planning and testing by pointing us in the right direction.

Great video, thanks for the insights.

Brilliant! You describe the main problem with AI code tools very well, I have the same experience.

Knowledge base is an apt description I use ChatGPT for. As a game/digi twin developer, using Unreal Engine is requires learning new parts of the engine every day, all of which is not possible to squeeze in to my little, squishy noggin. Questions would range from project specific real world domain knowledge, to questions on how to implement something small in C++, how to use or extend UE's API. Most of the work is still done by me, but getting some consise info on a topic and validating that info is a lot faster than getting that consise info myself. If it's a more complex and or niche 3D tech topic, I usually browse though a couple of articles, tutorials and talks (Siggraph, GDC and such) on the topic, then ask a specific question to ChatGPT on how would X works in UE or how to make Y with Z in mind in UE.

Great video! Thank you

I totally agree, I haven't used it to outright generate code and I don't want to--that would subtract from the artistry of the craft--but every time I can't think of a good way to do something or I'm not sure how something will behave, I just chat with my good buddy ChatGPT about it and immediately make headway. It's also good for the occasional optimization pass, maybe there's a better way to do something I hadn't considered, otherwise, my ego is definitely stroked when the AI says my code is already optimal. It's not a tool to write code for me, it's a tool to make me a better programmer. Bonus: after a long day of coding, you can pop over to the sous chef module and discuss your ideas for dinner, learning about the chemical processes of cooking has been fascinating!

I was not a programmer, but using linux for a long time. Learned emacs in 2 months, using django for fullstack basically solving things i had no clue about. It would take me 3 years what i did in like 6 months. I solve everything by chatgpt, many times it requires me to understand the problem to get it right, so i agree with you. On the other hand I had no knowledge as you did, but i keep on pushing chat to give me possibilities to solve it and many times it does ;) Great vid tho