I started using co-pilot less than a month ago and I've already spent hours trying to work out what on earth was going on ... copilot wrote code. So 100% with you, boilerplate code is freaking awesome, just gets the fluff out the way quickly so you can get down and dirty. But don't let co-pilot code :D
As junior I'm tring to use it only to fill variables and write comments faster. When it suggests me whole bunch of code sometimes I read it or just ignore it. Like chat gpt it gives some ideas but I would not trust it enough to just copy paste.
Realistically, we often aren't giving it enough information to make good judgments - it has to guess from code we're writing, what we really intend. Comments do help in creating context. And, of course... more powerful models are coming (though they'll likely cost more).
Working with these LLMs is like managing a team of brilliant high school interns. They're brilliant, but they're also teenagers. Manage them wisely and they'll multiply your abilities (at least within some narrow domain). Let them walk all over you and they'll write garbage faster than you can pick it up.
You are overestimating these tools. It is mostly like training a parrot and then asking him to write your code. Both Parrot and AI don't know what they are doing but they are pretty good at repeating things.
You know how the worst part of your job is combing through a junior dev’s PR to make sure they aren’t going to break everything if their code gets merged? Now with CoPilot you can have that experience writing your own code too!
I love that part of code reviews. Once I'm on a roll to find potential issues in a PR I don't hold back. I once had a PR with 400 loc changes and I must have put a comment on every other line, sometimes multiple ones. The best part is seeing them do something with that feedback. More senior engineers tend to think they know better and cannot be steered to a better solution without a lot more effort.
Having spent a few months letting copilot write bugs for me, I'm no longer worried that it's going to take my job. Also, adding copilot to cmp seems to be the better option for me since it's easier to ignore bad suggestions. Love the video, been waiting for the review!
copilot is the best if you're using it to make development fun. If you let it take care of filling in the boring things like lists of constants, filling out predictable arguments, filling out syntax that's easy to forget, you end up having way more fun writing out the logic that requires actual thinking. If you use it to fill in logic, even if it did it correctly 100% of the time, you're going to get disillusioned with the whole dev experience and you won't have any fun feeling confused and like you're not leveling up your skills at all.
@@JFrameMan I think hes burned out developer and he meant that because you still have fun writing code, you must be new into programming... dont listen to him, programming is fun, just dont get stuck on one position for too long...
3:43 There is actually even a better way to prevent this bug than adding the condition: Instead of using a condition you can match against the slice of self.frames (match &self.frame[..]) and then use [first, .., last] as pattern to guarantee that first and last are not the same element, while being both more readable and less error prone.
Using it for a month, I really enjoy it writing boilerplate for me, but I enjoy it more when it writes unit tests for me!!! I've got some whole suit of tests entirely written by Copilot with few fixes from my part!
Yeah, I've found this, too; copilot is really good for spitting out unit tests for your code. It even comes up with test ideas I should have thought of. I'm just like tab tab tab tab tab tab tab thanks for the 6 new tests
Copilot has really improved my scope in terms of backend programming. All helper functions I need are auto generated, even the ones I didn't know I needed. Also in terms of translation of a requirement I copied somewhere or a json I need to manually write
I find that it's good for rewriting existing code. E.g. a couple days ago during a Scala code review, I saw someone had a long chain of matching on Optional values, and I wrote `// instead we use a for-comprehension`, and Copilot perfectly rewrote the code as a for-comprehension.
Coming from the opposite end of the spectrum (i.e. noob coder), I've been using copilot to write simple 1 or 2 page python scripts, eg for api pulls and sorts. For basic 'get a task done', where you're not concerned about quality coding, just getting a simple job done, it's been a dream come true. Yes I hit many errors, but not the type of issues I used to have that would make me quit and figure out a non-code manual way around my task. The gamechanger for me is I don't have to read several books, watch hours of tutorials, and become an expert at troubleshooting to get simple things done. AND yes I do learn a lot of python commands and syntax tips from CoPilot in a much more effective and faster way than I would through standard learning methods. In fact when copilot gets something wrong it's sort of a game to figure out what it's suggesting vs what you're trying to do.
The fps function should use (n-1) for the calculation rather than n. Consider having 3 frames, one per second. So the duration difference between the first & third is 2 seconds, then you're dividing by 3.
Using it for a week and I love the autocompletion so far, no amount of vim skills could save you that many key strokes, like writing arguments + types a have never been that fast, when I'm doing something obvious/easily understandable and it spots it or the pattern it's also cool, it takes some corrections but it's faster than from scratch. I still have a key mapping to toggle it tho since it can be annoying if you're focusing on doing something non trivial and it proposes you nonsense.
My experience exactly after 4 months. My take is that you code faster in the beginning but the gain becomes less and less. Coding is much more tiring because you don't have to write the stuff that require less thought. You're always working in the hard bits. It won't replace good programmers. It will reduce the number of good programmers because beginners will rely on it.
Wrong. The "hard bits" are precisely the precious, _creative_ bits. The part that the most productive (and precious) programmers are more _willing_ to do. (like doing your pushups vs watching TV😄)
One other thing that Copilot is really good is generating code for an API that you're unfamiliar with. I had to do some stripe scripts to check that all invoices were generated correctly. It would take like 30 minutes of me looking at the documentation, working through all that text, instead Copilot generated the code for me in 3 minutes. But it was still important that I knew some details on how the API worked beforehand, so I had the opportunity to fix some of the bugs before they happened. If I hadn't knew the Stripe API details beforehand, maybe it would take like 10 minutes. Still better than 30.
I think this is where it's most dangerous as well though, especially if one starts to do this regularly. Imagine it writes some code for an API that you don't understand, you haphazardly review it and then there is some subtle bug that isn't caught. I wouldn't want to let copilot write any code for me for which I don't already know what it should more or less look like beforehand. That being said, I think it can be really useful in quickly getting to know the API, and writing some rough prototype code.
@@aure6898 I definitely think the same. It's good for early API exploration or obvious/unsurprising code generation. Actually, since I made that comment, my perception of Copilot's usefulness has been reduced significantly. It seems to be good only at doing the obvious (pattern repetition, copy pasting, etc) and requires a lot of guidance in most places. In my compiler, it just doesn't understand much and resorts to dissing my code by saying "this is a hack but should work for now".... frankly, Copilot is not wrong there lol but it's just not that useful.
The summary is my exact experience of copilot. For languages I consider myself competent in, I love copilot. It helps smooth out the boilerplate and I generally can provide it with enough context to guess what I'm up to. For languages I'm learning, I generally turn it off and debug interactively with gpt
Honestly the best thing copilot has done for me is speeding up writing unit tests Not only will it take out any of the boiler plate around, for example, mounting components, but also, if you give a good enough name to your test case and it is a simple case, it easily create it for you
@@bdkamil95 no they are not, you are using a phone that was most likely made from child exploitation in China yet I am not blaming you here for it, but Apple or Samsung, or whatever phone manufacturer you are going with. You can't blame the consumer because they can do little about it other than protesting, besides rather than being unethical it is just of dubious moral, it isn't as simple of a problem as people make it seem
I find that for C# Unity development, it's really good at filling in the boilerplatery stuff, like the caching of get components. It's also good at knowing what kinda stuff you'll likely do with certain variables, probably because gamedev code is so similar a lot of the time. But yeah, I only use it to fill in a line of code, but still, I use it quite often and would probably feel the pain without it.
Unity dev here too, I use it the same exact way. One line at a time. I also find it incredibly good at writing SQL queries/strings, as it knows what I want to fetch and can just know what my database schema looks like somehow. Try using vscode on a plane one day, you’ll find yourself typing then stopping to wait for the auto complete and then realize a few moments later it’s not coming 😅
@@NicolasSouthern Okay, Mr. 1000 lines of code each day. Maybe you should disable auto completion or even better remove all your so-called productivity tools.
@@rentefald nah, I’ll keep using them. These productivity tools assisted me in getting a product from theory to market in probably half the time and it’s currently generating a solid income for me. I don’t need to write every for loop to feel good about myself. Thanks for the suggestion though!
@@rentefald this is a tool that is like a power tool in the hand of an experienced woodworker versus a power tool in the hands of an inexperienced woodworker: the experienced woodworker will get their project done faster, the inexperienced woodworker will likely botch their raw lumber that much faster than using hand tools, and may take off their hand in the first place. I find it comical someone in the field of automating things as the job description is screaming at others to avoid automating the task of automating things as much as feasible: it's the epitome of extreme hypocrisy personified.
Another great use for Copilot is inline documentation, like in C# you can document functions with XML and I find that Copilot is a treat in those situations, and if the comment is invalid you just CTRL Backspace a bit and correct it.
Super helpful video, I've been using Copilot myself recently for the past month and have had the feeling I spend more time reading and rejecting bad code than I save from accepting good code. Love the idea of just rejecting the logic from the get go.
Exactly my experience as well. I love copilot for taking over the boring work for me, so I can spend my time actually tackling the difficult stuff. Writing boilerplate, filling translation files, mapping data from one structure into another and interestingly documenting my code are now all things I spend less time on, since copilot usually gets these things done for me.
Great video. Hit the hammer on the nail at the end when saying it basically works extremely well when you are already at the top of your game and just needing to smash out boilerplate. Personally I would add one suggestion/clarification to the last section. If you're a junior, I would also highly advise AGAINST using copilot, but I would extend that to any level of engineer that is starting at a new (existing) company, and coming into a codebase they have no knowledge of yet. Knowing the ins and out of the codebase you are in plays a big factor in how quickly you spot copilot's mistakes/inaccuracies. Other than that Awesome Vid!
If you are at the top of your game, why are you writing boilerplate? All that means is that you haven't been able to create an efficient environment for yourself and your team.
I strongly dislike a bunch of unnecessary boilerplate, so I'm genuinely not happy that copilot makes it easier to just generate a bunch of extra boilerplate so easily. IMO, it's _supposed_ to be annoying to type it all out, and this is usually a good signal that perhaps a better design is needed.
@ThePrimeagen not related to the video at all, BUT I just discovered "The Last Algorithms & Data Structures Course You'll Ever Need", and I'm so fucking excited. I've learned this stuff several times, but I feel like everything kinda leaves my mind if I'm not practicing... and it has been some time. Getting ready for job interviews, this feels like a godsend. I fucking love you man. Thank you so much! Keep doing what you're doing brotha!!
-Out of context You really made my life a bit better, I'm not in my best moment, but you give me the motivation enough to keep going cause you're a "real" whatever that means, just want to say thank you dude.
What I find really usefull is wiring unit tests with copilot. Have you tried it? This works really great if you have existing tests and want to add more cases. Just name the function properly and copilot does most of the work :)
Copilot needs to write the test first, then once they are reviewed, then write the code to make the test work. This will remove the bugs. Since the responsibility is back on the developer to ensure the semantics of the system behaviors are well captured in the tests.
Thank you for taking the pain for us all! I've been thinking about jumping into using Copilot but honestly I don't write all that much boilerplate in my current job.
Love this so much, been using Co-Pilot and had a few moments where I'm like "No wait, that's a bug!" The biggest difference between me and you though is that when I used Co-Pilot, it had actually wrote code for an edge-case and I created a bug by trying to correct it 😂
I started writing mongoose schemas, and its suggestions were great. When creating a few specific schemas, it even suggested fields that slipped my mind. I loved it for that. However, that's all I used it for, so far. I've been making sure my data models can handle the feature set I want before coding anything else. We shall see what happens when I actually get into things.
This is basically how I have been using co-pilot, mind you I'm a student, and copilot is extremely useful for redundant tasks, basically copy-paste on steroids. I once let it write some assembly and boy did not work out well.
Nailed it. 😂 I use it mainly for line completion, boilerplate, and on occasion, it will surprise me with a logical suggestion. $10/mo likely makes me 10% more productive.
Totally agree with this take. I'm a full stack web dev and have also been using it for about 6 ish months and have come to very similar conclusions. I will say that for me it's worth the subscription fee just for the efficiency gain I get from it auto filling boilerplate. And every once in a great while it does spit out some logic that is actually sound...
@@ThePrimeagen i watch your videos to chill and lighten my mood, good to turn the brain off and watch an entertaining but enlightening shortish video now and again. Compliment, and probably good for the add revenue, that's all :)
Def agree with this. I feel like co-pilot has been great at typing out things for me that had pretty clear in-context "templating" provided by me, but I would never trust the code it actually provides on its own. It's also great if you're just mentally blocked and you just need some kind of starting place to work with.
Same experience. For autocomplete, boilerpate type stuff it is good, but for logic it usually gets it wrong if beyond something simple. It definitely helps but I l am not worried about anyone losing a job anytime soon.
I know you really liked the DTO functionality of co-pilot, but almost all existing basic intellisense can already do it, just thought I'd mention that, love the vibe of your video, quite informative, thanks.
Right, GH Copilot is good for autocompletion, and doing some boilerplate stuff like wrapping hardcoded strings in translateable components. It's exceptionally good for writing tests for existing functions or components, saves so much keystrokes! However, you should always skim through the suggested code and do the corrections, because it suggest erroneous stuff pretty often, and you can't just blindly trust it. Just as it was pointed out in the video. Overall, GH Copilot can significantly improve your productivity by reducing the amount of typing, but it cannot replace a human coder (yet).
The Go LSP takes care of a lot of boilerplate without having to use any fancy AI. I just start initializing a struct object, double activate the suggestion keymap, and all the struct initialization lines are populated. And the best part about not using an AI to write my bugs is that I get to write them myself.
I started playing with wgpu in rust recently. And there is so much boiler plate. After a few hours of working on that project copilot just started smashing out those texture descriptors, buffer descriptors and so on. It saves you a lot of time, but only on the stuff that's annoying about coding. If you are not exactly sure what you need, it most likely won't do a good job
it's also great at translating my logic and teaching me the proper syntax when given a large codebase. i'm surprised at how quickly it can learn quirks like that
Converting data structures across languages seems to be my favorite part. At the moment I am using Copiolet to convert many json schemas over to rust types. All I have to do is paste the json schema at the top of the file. Once I do that it just fills in all of the comments and data structures, after a bit of prodding at times it gets it done. I have found this the biggest time save to be honest. It's not perfect at this, but most of the time it can do the simple repetitive work of copying over tidbits of information. I still have to have the schema open on the side due to simple mistakes / changes in implementation. Overall, after using it since its release I am pleased at the speed up in development times. - Note on json schema conversion Yes, I am aware typify exists, but it isn't perfect, and I have run into many issues using it. I did file some issues on the repo so the maintainer is aware. Along with the code generated being a bit of a mess to read.
I feel like I'd rather just write my own code than babysit a junior developer (which is kinda what this feels like to me). I think the tech will get better and it will eventually reach a point where you basically have to use it. But for now I'd rather not.
One of the absolute best uses I've found for copilot, is writing tests, especially in rust where you write the tests in the same file so it can read the source material.
CoPilot should have a partial application feature where it understands what you want to keep and then get rid of the unneeded code, making sure that things are still valid syntax (e.g. don't delete end braces).
@@hannessteffenhagen61 Both models are based on GPT, but CoPilots model has also been trained on source code from GitHub (and likely other sources) so it has a bias towards code. CoPilot also uses additional prompts internally to get more code-oriented results. They have a CoPilot Experimental plugin with additional features, such as refactoring tools and explanations from code. It's not unfeasible that they'd be able to construct the prompts to produce this kind of behaviour.
Got it, using Copilot to generate boilerplate and types. We still need to fix the code logic ourselves, at least Copilot gives us directions on where to start.
On your second point, I find that my copilot only RARELY recommends more than to the end of the current line. Which I do like, since often times the rest of the line is very obvious. If I want more lines, it will recommend a next line after I go to it, and so on. I use it like an actually smart auto-complete. Like context aware fill or something.
It'll be interesting just how much better it copilot-x is with the GPT-4 model and added features. I've gotten AMAZING use out of having it write scripts for me that deal with APIs I'm unfamiliar with/other things like that. Finding all the little things add up and take FOREVER where as validation of endpoints used in a script is much faster. As well, having it write tests/boilerplate for code is sublime.
Having to re-read generated code, understand it and verify that it didn't introduce subtle bugs that you would never have done because you know how the codeworks contextually has to take at least the same amount of time that it would writing your own code. Additionally the chances are higher that you then still know what you wrote two weeks ago!
True, although sometimes copilot's code is truly brilliant in a much more concise or expressive way than mine. Other times, not so much. It's like pair programming with a brilliant but drunk colleague. I've actually learned quite a bit from it.
1. I agree juniors definitely shouldn't use it. But with that in mind, I don't agree with your first point on not using comments to help. If you're using comments to write entire functions for you, then you're likely a junior. But sometimes, copilot just can't pick up what you're putting down without some context. I most often use comments within a function when I don't remember/don't know exactly how to use a third party library. Particularly if I'm interacting with AWS stuff, copilot can easily fill that out for you, but you need to provide more information than the function name/params.
I see these tools as being akin to a map vs a satnav. using a map builds a mental map in your head and you become less reliant on it as time goes on. This to me is like reading docs and learning the tools. A satnav just tells you where to go and unless you explicitly try to learn from it most of the information it gives you goes in one ear and out of the other. It does make me worry about programming literacy going forward and i agree with your assessment that it's probably not worth it for juniors, and even more experience devs should be careful
Yes this is a potential problem as even if the tool produced 100% accurate code and logic, it would result over time in you not having much understanding of what your code is doing whereas when you write it yourself you have a more clear picture in your head about what it is doing.
Had approximately the same experience after 24h of using it but was thinking I was using it wrong. Thanks for your video. A good boilerplate assistant but not sure it's worth the price.
I am 1 day into having CoPilot in my JetBrains IDEs... and it is, for sure more than a snippet generator. I have so much to learn, but giving it project context has it returning code that looks like I wrote it, albeit sometimes/often quite wrong. I felt I was missing a place to prompt... but, I guess doing what I was doing yesterday, by deleting the wrong parts and letting it continue to suggest may be the way. Day 2 now, I guess I have a side pane with other "guesses".. I'll see how that works. Thanks for the hints on how to get it to do the right thing...
Programming is writing explicit instructions to the machine to produce a result. Using AI for programming is writing explicit instructions to the AI/machine to produce a result that you'll need to validate/review/change before sending the instructions off to another machine. Haha, I've just not bothered with AI until it gets better, just feels like more work. ChatGPT can be great for quickly generating PoC code and specifications/requirements though, but not needed in the editor.
I just tried copilot today writing a varint encoder/decoder in Typescript. I agree with your assessment, and I would also like to add that its really good at writing JSDocs if you ever make something public facing in Javascript.
I like working with copilot. I have 20+ years of experience so I am able to verify its code and tell if it's OK or fix it. Or prompt copilot to fix based on my hints. Yes, it makes mistakes and can be annoying. And yes, it's great to semi-auto-generate repeatable code like entities or queries. But it also can give great ideas for solutions or fix my mistakes. Overall, I like it. But I agree - I wouldn't recommend it to everyone.
i agree this is how i use copilot too, sometimes wish it was a setting to turn off those long suggestions. i also usually don't do the "comment driven development" thing BUT i do find that it is useful for writing unit tests honestly
You can ctrl+right to accept suggestions word by word instead of as a block. I use it quite frequently bc it's annoying when you just need some boilerplate and it thinks you need a whole 15 lines implementation of the thing
I mainly use it to do all the tedious stuff like autocompleting several nested brackets or parenthesese and generating boilerplate. It's really good at picking up patterns which comes in handy many times.
The more you know about what you are doing and what you want, the best the collaboration with such auto-completion AI tools. I think it's just the beginning though, and soon enough we will see much better code generation and UIs to use them more efficiently.
That’s what I thought as well and here I was am I using it wrong ???! 😂😂 Great video and most importantly a no BS AI will replace kind of video as well ❤
I strongly suspect the logic correctness of generated code depends heavily in the preciseness of your prompts combined with the corpus of code it trained in in the language in question. Thus far (not even 2 days use, admittedly) it does pretty well for Swift for logic, if I describe things precisely enough, which I'm seeing how well it works for comment-driven development (not sure if you coined that term first or not, but it fits!) of a solitaire game of my invention which has only before been implemented in Objective-C, and never escaped my devices. Conclusions thus far: 1. It uses an older version of the LLM than Bing Chat, and Bing Chat is clearly superior for creating logic from descriptions. I've signed up for the beta of Github Copilot X, which will have GPT-4, which Bing uses. The problem with Bing Chat (other than inconvenience of copy/paste) is it insists on you moving on even if it generated other than you hoped. They wish to steer you towards Search, which is reasonable for something you don't pay for. 2. I've already seen some truly dubious generation for options, as seen below: var suits: Int var ranks: Int var deck: Deck var stacks: [Stack] var score: Int var selectedCard: Card? var currentStack: Stack? var currentCard: Card? var currentStackIndex: Int? var currentCardIndex: Int? var currentCardScore: Int? var currentStackScore: Int? var currentStackScoreIndex: Int? var currentCardScoreIndex: Int? var currentStackScoreCard: Card? var currentCardScoreCard: Card? var currentStackScoreCardIndex: Int? var currentCardScoreCardIndex: Int? var currentStackScoreCardScore: Int? var currentCardScoreCardScore: Int? var currentStackScoreCardScoreIndex: Int? var currentCardScoreCardScoreIndex: Int? var currentStackScoreCardScoreCard: Card? var currentCardScoreCardScoreCard: Card? var currentStackScoreCardScoreCardIndex: Int? var currentCardScoreCardScoreCardIndex: Int? var currentStackScoreCardScoreCardScore: Int? var currentCardScoreCardScoreCardScore: Int? var currentStackScoreCardScoreCardScoreIndex: Int? var currentCardScoreCardScoreCardScoreIndex: Int? var currentStackScoreCardScoreCardScoreCard: Card? var currentCardScoreCardScoreCardScoreCard: Card? var currentStackScoreCardScoreCardScoreCardIndex: Int? var currentCardScoreCardScoreCardScoreCardIndex: Int? var currentStackScoreCardScoreCardScoreCardScore: Int? var currentCardScoreCardScoreCardScoreCardScore: Int? var currentStackScoreCardScoreCardScoreCardScoreIndex: Int? var currentCardScoreCardScoreCardScoreCardScoreIndex: Int? var currentStackScoreCardScoreCardScoreCardScoreCard: Card? var currentCardScoreCardScoreCardScoreCardScoreCard: Card? var currentStackScoreCardScoreCardScoreCardScoreCardIndex: Int? var currentCardScoreCardScoreCardScoreCardScoreCardIndex: Int? var currentStackScoreCardScoreCardScoreCardScoreCardScore: Int? var currentCardScoreCardScoreCardScoreCardScoreCardScore: Int? var currentStackScoreCardScoreCardScoreCardScoreCardScoreIndex 3. You need to keep it from trying to do too much at a time to keep sanity. 4. Fully agreed with your assessment that inexperienced developers should not rely on it to generate their code: if you can't write the code without Github Copilot, you shouldn't allow Github Copilot to write it for you, as a result of the statement below: Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan 5. Never accept code you can't read that isn't clearly correct, or correct enough you understand how to change it to make it correct for your purposes. Just because code may compile, doesn't mean anything beyond it compiles. 6. If all it can do for you is create all the boring boilerplate code and test suites, assuming you're not just writing a meaningless amount of code per month, it's easily worth the current price: you can easily get the $10 ROI in less than an hour of time saved per month, especially if the generated code for your language doesn't have syntax errors and typos, and is correct to start with: how many stumble over typos? Hopefully your regular IDE will show the issues before you get past the identifier or keyword so you can fix it quickly, but that can reduce your flow time percentage.
Great video! I program in several languages with different IDEs (VS Code or Jetbrains, actually) for production and teaching. The little I have used Copilot has matched your experience of boiler plating. But it makes me wonder: the example with filling in the Media type with Typescript is something that Jetbrains does with syntactic analysis. For this, AI seems to be the long way to the goal. But I find that VS Code is lagging behind substantially on syntactic analysis and try to compensate with AI. Does that match your experience and is it the better path?
You could be right. There are some things that it does so well. Like at the end with auto completing audio FPS. It can get pretty dang fancy with logical boilerplate
