Hot take: Code-writing AI is hyped up so much because companies needed an excuse to lay off many people, and then hire them again later as independent contractors with far less pay and benefits...
I use claude and AI very effectively. Like right now I am writing a Kotlin app, but I only know javascript. But you have to know what you are doing to get real functionality out of it. I read every line of code and I am starting to really get Kotlin. But it's all about the pseudocode. You have to think through what you want and communicate it very clearly and expect to have to hone your communication several times. All of that makes me a better developer. No AI cannot replace me, but it sure does help me.
@@patrickjreid yes AI can definitely help. But does is really help more than looking the info up in Wikipedia or Google Search? I doubt that a little bit.
@@bpscast For me AI helps a lot more than Google + StackOverflow or Wikipedia. AI allows me to be faster in getting the information and/or suggestions I want, and if the information/suggestions are wrong I can quickly figure that out in my dev environment and iterate from there. Now even though the traditional method of Google + StackOverflow has less hallucination tendencies (sometimes the info is still wrong/outdated), there are multiple times when I used non-AI code from someone's SO post or blog and it didn't exactly fit my scenario, so I had to iterate all the same anyway. In any case AI has already given me a huge productivity boost in my development, and the AI models themselves plus tooling around it are only going to improve. I've been trying out Cursor for the past week, and I can see my productivity improve even more with some of their features. E.g. I used their chat function with context to code a new feature for me, and I basically acted as a code reviewer. Even though Cursor didn't get it right the first time, after one iteration with my own feedback it managed to get it right. Definite time-saver.
Replacing coders with AI is like replacing doctors with AI. Who really needs replaced with AI is upper management. Heard thinking, commonality, same old mistakes sounds like a perfect use case for AI.
It's certainly true that AI writes buggy code and "isn't there yet", but never underestimate the greed of corporations. Quality doesn't really matter to them. They'd be perfectly happy to lay off half their workforce in exchange.
been using Claude everyday for a side project. It's not the greatest at debugging issues. You usually have to hold its hand and tell them the exact few lines or functions to focus on otherwise it will overcomplicate things by default. What it's great for is generating some boilerplate or template for a new function or component. It saves tons of time from looking up documentation, googling issues on stackoverflow. You just have to read over the code it generates to see if it makes sense. If not suggest a different approach and it's generally pretty good at adapting since apparently you're never wrong...
AI is pretty good if you are using it to "query" stuff that is in documentation. in fact i often drop a link to node docs for an example then ask a questing around node that i know will be really well documented.
There is hope, but I'm not going to lie - it's looking ugly in the short term. If it's any consolation - the AI you might get replaced with will do a lousy job, and it will get fired, and I expect there will be a bunch of jobs for us later cleaning up the mess the AI made,
@@zokpls8712 Not yet - at least in most places. But some companies are starting to make pretty big moves in a direction that's bad for programmers (in the short term). BP says they've reduced their programmers by 70%. ( www.webpronews.com/bp-needs-70-less-coders-thanks-to-ai/ )
Models won't get a lot better, coding is just the wrong category of problems for Neural Networks, it's not hard to understand, programming is not a fuzzy problem, which is what they actually do well...
Models will get substantially better, lol. Don't kid yourself. There are already new focuses based on revamping AI reasoning for this specific reason. There are multiple papers by Anthropic, Microsoft, and OpenAI on this.
@@yds6268 The biggest way is to add new hardware that has higher perf/watt. Which is exactly what the new Blackwell Chips do, apart from that there will need to be significant changes in energy expenditure and/or management in the long term. Not something easy of course, but as AI is able to take on more tasks. That is an investment companies will be willing to do.
I dont understand why people are trying to crowbar LLMs into reasoning tasks. If they took the time to understand how they work, they’d stop wasting their time on this, and only use LLMs for things they are good at, which language stuff, not reasoning!!
Takeaways: 1. The code gens can at most do imperfect boilerplate stuff. 2. Coder monkeys are f***ed, and 3. All others will have more work proofreading and testing AI code. Fair assumptions?
I have been experimenting with the aider project recently. It has a few benefits for me, but big issues too. The biggest benefit is that it changes the dynamic of programming from pure creation to creation&collaboration. It creates mediocre code pretty often (backed w/ Sonnet35 or gpt4o), but it gets close enough that I change in the editor and move on. I'm prompting it extremely granularly too. 'add a new arg here', 'write a test for this one method', etc. If you try to do too much at once, everything falls apart so quickly. Also I'm a senior dev who can nearly instantly read a chunk of code (in my app at least) and see things wrong. I can't imagine a junior or even most midlevel devs doing anything useful with this without hitting so many speedbumps.
agreed , aider workflow is good at small iterative additions and changes , id guess about 75% of the time it does well . the main failure cases are typically when trying to do big changes across a lot of code , and when it gets into a repeating cycle of alternating between 2 wrong answers .
The worst part about A.I that I've found at the moment, is that the API documentation that I usually rely on to get my day-to-day job done has either been purged or has been put behind an auth wall \ is not getting updated, in favour of "use our new A.I tool". - and the new AI tool is producing mediocre code at best or the purpose of me referring to the API documentation is to understand what a certain function, constant value or module does, and just getting an example snippet from a command line doesn't help with that, so I'm finding that I have to dig through definitions in the source code and take my best guess. Another part is what I would once refer to as an algorithm or set of algorithms within a given package or program are now being blanketed as "A.I."
I had to use image magic and convert recently on a Linux box. Darned huge number of hoops to get the SE to let me run it! I should have just spun up a VM, done the needful and sftp the results to the real machine.
Please never stop. This is such an uphill battle on something where it is so obvious. How can the sane people be the minority? Artificial intelligence should be the term used to describe the people who think the LLMs work well.
@@rasi_rawss Yes, and false advertising is illegal. You can't sell a product knowing it doesn't do what you said it can do. Look up Theranos. This crap is exactly the same and people should go to jail for it.
They don't care. As long as they can sell the illusion and make bank in the short-term, they could care less about the long-term consequences. I'm sure some of them even know that LLM capabilities are vastly overestimated and a correction will happen.
I think the biggest problem is the marketing angle where AI companies are saying that it will replace developers instead of saying that this is something that will make developers 2 times faster/eficcient for ex, thus reducing your development cost. In it's current state Claude AI is good enough to convince me to buy a subscription. Problem is that 20$/mo is not nearly enough to finance AI R&D, thus companies choose to overhype the AI so they can fool VCs into funding them with insane amounts of money.
I doubt it will make devs 2 times faster I have used gpt4 but it's garbage, it generates really bad code that will cause problems in the future and it doesn't understand how to fix it when it is integrated with kinda complex codebase. Sure it can write boilerplate code for already popular libraries but you can easily google that.
I asked Claude to create a Symfony (PHP) command to do simple checks that language binds of some forms existed in their corresponding language files. It only imported 3 classes that it never used, 5 variables that it never used, called 2 methods that do not exist, and used 1 method that was deprecated in 2018. It did not perform the check in a workable way and reported everything as an error. I've never wanted a junior developer who can't learn, but that's apparently what our team just got.
Didn’t we go through the same hype cycle with self driving cars? The usury based financial system gives all the incentives for creating bubbles before they burst and the big companies are saved with taxation or inflation.
When *AGI soon* reached pitch fever middle 2023, I used to post the following comment, and got spanked for it: "Billions of $ + top talent + infinite amounts of data could not crack Self Driving Tech, but here we are, hoping AGI, something infinitely more complex, will be solved. What am I missing?"
We've been through this cycle countless times, as you say we've had self driving cars, the metaverse and NFTs, no-code replacing programmers, 3d TV's, cryptocurrencies, and a tonne more I can't even remember, and every time its "no no its real this time". Oddly every iteration seems to be more outlandish and unrealistic than the last, looking back on it 3d tv's seems almost quaint compared to what we're at now.
Self driving is pretty decent for highway driving, I use it on my Tesla regularly even though it's "enhanced driver assist" at this point. I expect LLMs to be the same, more as assist features, till we are able to take another significant leap in LLM tech.
@@maheshprabhu Let's start from here: the definition of "self driving". The tech is not ready, as many RU-vid videos show. I am not trusting my cat inside those cars when the "self driving" thing is on.
A project I'm working on started using AI code reviews & it only caught 3 issues, and 2 of them were for completely wrong reasons. All 3 of them were incredibly basic errors that I would have caught without it just by testing it once I was done setting up all the scaffolding. The rest was dozens upon dozens of misunderstandings of the code base or regurgitations of suggestions that read like they were from a cargo cult programming blog. It took me 30 minutes to sift through utter crap to get very little benefit out of it, and it was only that short for the amount of reviews it gave because they were so crap I could dismiss them out of hand. It's not even like my code is perfect either. My previous PRs on that same project have gotten plenty of meaningful reviews and adjustment requests from other human reviewers in the past, but the AI didn't catch anything of that nature.
Claude can write really impressive code if you babysit it every step of the way. It will also destroy your code in really impressive ways if you don't and just blindly follow its instructions. For example, it had me merge two unrelated functions into one when I asked it to print an existing array as a debug line if there was an error. Not only did it end up not printing the array but its next suggestion was to split that function.
AI Gen Code, including Claude is just a tool 🧰. It's better than being stuck or being on a blank screen. It's better than just rubber ducking or random Googling. But, it's just a tool 🧰 and without trained operators, the tool 🧰 will not work so well. Clueless tech business leaders that are de-valuing and/or mistreating their best devs will realize this in 2-5 years. ⏳
Great one. I was bracing for you calling out mistakes I myself might make but nope, the demo and test were indeed garbage. Thx for these sane well-reasoned takes.
And to explain algorithms, give ideas for algorithms, come up with APi's and libraries you did not know existed, be a better intellisense, right little utilities, critique small bits of code, suggest names for functions, find duplicate functions, write simple unitTests, search through a lot of data and analyse it, suggest fixes for problems quickly without going to stack overflow.
@@whig01 It doesn't make sense to say that a normal program is "effectively AI", just because it has some interactive options. That would make pretty much every app "effectively AI"
It's a fancy google search, it doesn't think or reason. What is the next most likely weighted token is the only thing that matters, and I saw a video where it very confidently said a Prime number was not prime, because the first token it matched on was "No" so the rest had to make sense. And it was confidently incorrect in it's reasoning (or the output was convincing to a human, no reasoning happened). Probably due to the fact, like you said, it's going on the most common response. And that could be a reddit thread with completely wrong individuals.
AI code assistants are used by major tech companies like Nvidia. They are just tools like but for sure they have made me more productive in generating efficient useful code but they make mistakes like humans. But as time goes by they will get better
Hello sir, I just want to ask a not entirely related question, but have you already tried and tested the ChatGPT-4o-latest (2024-08-08)? If so, how does it compare to Claude 3.5 Sonnet?
I have too noticed AI could not perfectly write code. It just helps writing simple code that programmer can use and write a bigger logic. though it wrote unit test good. Good use case of AI could be in testing
Its undoubtedly good at natural language processing, thats what it was made for, its really odd that we have this thing that can talk like us and just automatically believe that means it can do everything else like us too
It's Ok for coding. Definitely not great, but "not useful" is a bit much. You can rarely use what it gives you outright but it's very handy as a jumping off point.
Welcome to reinforcement learning, where the goal is to create AI, but without the I. To be fair, what did we expect? If you reward someone based on whether you like what they say, won't you just create the most perfect sycophant that only tells you what you want to hear? And that's all that these LLMs are - a correlation of weights and biases that link tokens in the query to the output tokens it should generate +/- some randomness to create the illusion of creativity. These "AI" solutions will always "provide" you with a "solution" and even write beautiful little comments, as well as insert magic bullet functions from libraries that don't exist. It's cute, but alas, a waste of billions of dollars and untold man-hours of labor. Big tech can't admit that, since then they'd be held accountable by the investors. It will crash and burn someday, - a correction always happens - just not in the short-term.
you come off as a pure pseudointellectual when you act like this technology is a waste when clearly you just personally haven't found use for it (or tried to?). it has made my job as a programmer faster and it has helped me learn many things in life outside of my job. You say it will crash and burn but it is currently making billions of dollars in revenue. If you have used the models for what they are useful for then you will know they aren't just an illusion of creativity, they actual reason about things in ways that haven't been reasoned about before explicitly and that's their whole utility. Even my company is using these LLMs for actual business use-cases where we previously required human brains. The costs are also incredibly low and getting lower by the day...
@@kyleyoung8179 No, append a trailing zero to 9.9, so it becomes 9.90. Now that we have 2 digit numbers after both decimal points, compare 90 to 11, and you see that 90 is greater. So we can conclude 9.9 > 9.11. The length of the string representations is such that len("9.9") < len("9.11"), but not the numbers they represent. A more interesting test case, that AI fails, is whether -2 is greater or less than +1. The correct mathematical answer of course, is +1 > -2. But depending on application, "greater" and "less than" may only be meaningful when comparing magnitudes, making what -2 represents be what most people would consider greater than +1. An example being electrical charge, where it is completely arbitrary what we call positive, and there is nothing "greater" about a sodium ion's +1 charge when compared with an oxide ion's -2 charge.
Most are getting this correct now. My 286 used to randomly crash in certain applications after I pressed the turbo button. Yes, computers used to come with a turbo button.
All these tools are basically goood summarisation of existing outputs from humans. For a very close question/query. They can retrieve a good summary. Nothing more, ask something a bit twisted, they will go complete kaput 😅
A.I. Coding and its hype will lead to digital Idiocracy. "Why does the crop now grow, we are giving it electrolytes" will become "why does the software not do what it is suppossed to, we are using A.I." But then again, I like idiots as competition...
The context is just too small still. I have an arduino code file thats only about 2000 or so lines, but it just cannot read it and answer questions on it. It just falls over and hallucinates loads. I was trying to get GPT and Claude to add some functionality and it just ignores whats already there and suggests duplicating functionality and ignoring the convention that is already there. I think for now Ai is only good for answering short small context questions. Nothing more.
I tried Claude 3 Sonnet and Claude 3.5 Sonnet for my own tasks (unit test generation for Python code, test generation for TF code) - I did not see any significant differences in the quality of the output. And no matter how detailed prompt I give, the produced test require ~80% of refactoring.
As soon as you said complicated ways to find prime numbers, I thought of the Sieve of Eratosthenes. I had the pleasure of coding that algorithm in ASM for a class project. I subscribed because it reminded me of, while tedious, how much fun it was to get to work.
I've been coding for just about 2 years only and have been using AI to learn most of the way. I do use Claude fairly often, but I just use it as a jumping off point and almost never actually use it completely. Especially for Svelte which was my last project, it knows a fair amount but the documentation and random blogs were far better resources
I appreciate your channel. The technical component of all this has been the part that I've basically had to start from scratch to understand all the bullshit going on. As someone with a journalism undergraduate degree I am embarrassed by how the press have basically been stenographers for these companies. It's crazy how terrible they've been in covering this.
I agree with you that even in 5 years, AI will never be as good as a human programmer, I think you can concede that in 5 years it will be so much cheaper than a human programmer, that hiring more than one for a given project will not make sense. As a teacher you know that college CS majors currently do a better job at coding than AI, but you also know that it also literally takes years to learn how to code at your level. Thanks to Retrieval Augmented Generation (RAG), you can literally train an AI to code At Your Level in just a few hours. That's the real threat, it's not what it knows, it's the insane rate that it can improve, and once it gets to the level where it can self-improve without human guidance, then it's game over for everybody.
I just asked GPT 4o: "Is 21577 a prime number?" - Answer "yes" (correct). Then I asked again: "Is 668887 a prime number?" (668887 = 21577 x 31). Answer: "Yes"
I tried this with my Excel spreadsheet with code I wrote for prime number detection, and got an overflow error when trying 668887. I revised my code to use long integers, and it properly detected 668887 as composite.
@@carultch I figured out that gpt generally has problems dealing with numbers. It tokenizes input and a number like 663837 is split into 663 837. Then.it has problems to comprehend the context because 664.837 are ultimately different things than number 664837...
@@breakoutgaffe4027 Trу swaрping Latin lеttеrs for a look-a-like Суrilliс and Greek letters with сompletely different рrоnunciations, and sеe if it confusеs the LLM. I did that in this сomment, and yoυ рrоbably didn't evеn noticе.
<a href="#" class="seekto" data-time="295">4:55</a> That's a pretty good summary of how outsourced codes have been coming back from the far eastern parts for 20+yrs. There is code, there are tests, they pass, except none of it is actually useful. Management loves it because it's cheap and fast.
Since im not a coder i had no idea how to fix a broken plugin for a gameserver. The plugin was abandoned and i did not find anyone who would be bothered enough to have a look at it except of one guy, who said he would if i'd pay him 100 bucks an hour. So i tried my luck with claude 3.5 and it not only fixed it for me, but it added new functions to it at my request. It is now working again, and i don't care if you think the code is bad or not. IT WORKS! And i did not have to pay a fortune for it.
Good for you. It is no proof however that Claude can code in general, it just shows that in your specific case you found a way to use Claude as a tool to get the outcome you wanted. That's great and shows you're really smart. Re Claude, News at 5.
One example cannot represent all situations. I believe that we should view problems from a developmental perspective. At the present stage, AI code definitely still cannot replace humans, but it can assist coders in improving coding efficiency.
AI is not magic. It’s a tool that can be used effectively or poorly. The IDE integrations are useful when used well. LLMs are also great for debug messages, sometimes (but not always) better than google search. They are far from being able to autonomously develop on their own.
You have been right about most things, but of course this is what this AI is all about, hype, a deflection, layoffs a 'necessary evil' since the dark objective is really about what most will understand as far as cybernetics is concerned, not robotics synthetic 'machines'' but of course this involved biology, even way back when they started messing around with neurology and publicly take what finalspark is doing and extrapolate on that. The darker objective is unethical, so whenever we essentially replace ourselves this will become clear. This is how they will achieve real AI or as they would say AGI, sure much of it will be mimicry but self-awareness, reasoning and emotions, the complexity of the human thought process spanning unrelated domains, all of that is a matter of course, the real question is, how close are they... 5 years, 50 years or are these things already walking amongst us and we are the proverbial Turing test. Regardless insofar as this, you are right, but they need to do this, fear and panic shuts down the prefrontal cortex and allows the limbic system to take over, crippling critical thinking, these things relating to 'ai' will fail, the current energy requirements to train the latest LLMs is insane so much so meta is (of perhaps this is done already) building a nuclear power plant for this very reason, a bit extreme, if people understand G P T tech they will get why scaling is a big problem... sure handy robots doing programmed tasks with some level of acting upon instruction from the user will continue and will improve but this GPT-like tech is likely the biggest most expensive con ever. I deal with machine languages (yes even MASM still) and also most of the others, be it scripting or otherwise, whatever I need for what I do, from robotics to SE and mostly now security (since that is what is affecting people the most)... I adapt to where my skills are needed. This AI code is crap, but I still thoroughly enjoy your exposition on it
ChatGPT: "here's your code for MSProject VBA" ME: "This is the 12th time I've indicated that you are providing Excel VBA code, not MSProject VBA code" ChatGPT: "you're right, that code is Excel code" ME: "If you were able to determine it was Excel code, after I pointed it out, why can't you simply pre-check the code before suggesting it"? ChatGPT: "er....uh....Context Management, Error Correction Mechanisms, Training Data Limitations...and btw, F U"
meanwhile, ClownStrike doesn't even bother to write a test to validate input length for code going straight into the kernels of millions of machines. Maybe they used A"I"?
I've personally found good results with Claude by restricting it to be a helper tool. Writing very simple things or giving me some idea of how I might approach that almost like discussing with a colleague. I keep in mind that they are LLMs and only predict the most likely word to construct a sentence and have no concept of logic or design. For example using them to understand or create regex I have found them to be good. Manipulation of data in certain scenarios and writing very basic functions which aren't complicated but take time to write for example mapping data. The minute you ask it to go outside of this scope it isn't worth it
I know they're hoping and praying that if they just keep throwing more money at the problem, eventually all the problems will go away, but... at what point do we just admit that we're falling for the sunk cost fallacy, and it's cheaper to just pay human beings to do the job?
Claude is generally very good at coding and picking on a bad demo doesn't change that fact. People often get a few views on their video for a topic and then get tunnel vision and become a one trick pony. It's fine point out a demo that's wrong, but to then generalize it as if this is a typical result when it's not (actually it's quite rare for Claude to give you just blatantly bad it wrong code) is wrong and misleading your audience. Im a dev of nearly 20 years who uses claude daily. I suggest you actually a) take the time to learn how to integrate it properly into your dev work flow, assess it by actually using it in real world scenarios on top of what you're doing here, in order to develop a more well rounded perspective. Or not... There's plenty of views and money in AI hating.
The only place I sort of trust the output of an AI is in one off short python scripts, but even then I usually have to change something. When doing professional or "serious" work I don't find myself using a chatbot all that often. I find that it's pretty good at pointing me in the right direction, but the code itself is not very good and accurate most of the time. Also having used chatpgt for a while and now claude 3.5 for a while, I'm considering going back to chatgpt. People say claude is better at programming, but honestly, that has not been my experience. It seems to missunderstand my questions a lot, regurgitate the same code I've given it and other things. It could be that I'm just more aware of these things and I'm going to have the exact same experience with gpt4 though..
Sadly upper management still has a goal of firing 60% of the engineers at my company within the next two years. Of course AI can't actually do our jobs but they may only learn that after the fact. It's demoralizing working for people who don't understand what you do and see you only as a cost center.
are we in the same company? our CEO (and CTO) started preaching laying off people with a hope that 30% of engineers would be replaced by the "AI". It shows how brainless management is.
My feeling is that the AI pair programming thing is a little like the 'DreamWeaver' craze of the late 90's early 2000's... it looks like a winner, but of course you have no clue what all this code is if you aren't a coder and would leave it in there, a lot of it has security issues too! The AI will never remind you to run unit tests or stress the code at high load, thats all up to you and your knowledge of coding, and if you have none then you are like a bull in a china shop... So as a tool I use the pair coder a lot, cos its got all that stuff I used to copy, paste and modify from stack exchange... but its faster. However having been a coder since the 80's I KNOW WHAT I'M DOING!
I've heard somewhere that github copilot sure was making coders faster, but also brought up code churn rate (rate at which a line of code had to be changed). Might not be causation....
our CEO is all behind AI , its so depressing any meeting he literally just brings up chat gpt and say sees there is the answer like yea its *an* answer and it may work but its not that simple. feel like just uising Chatgpt all the time and not putting any thought in to it, as it seems like thats what people expect. its depresing and lame and sometimes these models dont even generate the right answer like if you asked it something it may generate it right but if you asked the same thing 20 times it will give some variation that is just wrong its all just random.
You make great points, but i feel like you are missing the bigger picture. The current models are extremely primitive. You do a great job of highlighting problems with them. Future models will be trained with knowledge of the type of problems encountered and how to solve those problems. It will be an iterative process of improvement. How long the process will take is up for debate but I expect that most problems will be solved in the next 2 years. The fact that the output of LLMs is stochastic is something I see as a positive, not a negative. I want my models to compare different ways of writing the same function and to learn what works best. I am working on a coding assistant and I will use videos like this as guidance.
AI maths is an issue because of tokenization, which is a perf shortcut. This problem should mostly go away with the next generation of models. Or failing that the ones thereafter.
You're getting replaced by ai just like every other profession. There's no stopping it. Ai art generation was just as goofy and laughable when it started. Pointing out that it's not perfect yet won't change this. The future of programming is creative types innovating and number crunchers won't be needed, just like artists. Ironically the idea guy will be all that's needed 😅
As a Full Stack Developer, I completely understand and agree with you. However, this field will eventually improve with time. We are coming up with new ways to improve models every day. We are still in the early days, and much of it is mostly hype. To be honest, I pay for these LLMs and use them too. They work better for writing, but for coding, it's still too early.
A lot less egregious than devin, but fair assessment. Albeit a lot of these issues could be solved or mitigated to a large degree by using something like typingmind with sonnet 3.5 + perplexity for example--to look up the most current and/or correct implementations. Claude 2 to Claude Opus to Claude 3.5 sonnet were all huge jumps in capabilities. Lets see what happens with 3.5 Opus. Lets also see if any of the reasoning changes actually pan out anytime soon.
<a href="#" class="seekto" data-time="755">12:35</a>: _"The bubble will pop"_ the bubble popped already the last week when the stock market fell, and the market analytics called it a "reset after the AI hype".
