Snip taken from "Preventing the Collapse of Civilization / Jonathan Blow (Thekla, Inc)" Original video: • Preventing the Collaps... #gamedev #gamedevelopment
I feel like his argument is misplaced. It's like saying "back in the day, you could just take a few logs and build a house in one day". Yes, and that house wouldn't have plumbing, electricity, sewer. It wouldn't have good insulation from the outside. The roof would leak a bit when it rains, making it impossible to store certain types of possessions. But hey, you'd have a house in a day. And on the flipside, you can still make simple things with simple computers, and people are still doing it. Grab an Arduino, flash it with whatever you want, and you can write your own OS. What's that? Oh, you want more than 64K of space, the processor is slow, and there's limited support for interrupts and no way to play realtime video? Well, welcome to Unix 1.0. Standards have changed. Programmers are also less productive, but I feel like he's getting further away from the "why" by looking at requirements. Whole lot of people are chasing the money these days, coming out of two week bootcamps, instead of having an actual passion for writing code.
I would argue it's the modern houses that leak and last nowhere near as long as the old more primitive ones. Case in point... I have windows software I wrote 25 years ago. It still runs today. Try that with today's tools. Another cracker feature... Real time remote debugging that was 1997
I agree and I think a lot of the "lost productivity" is actually being wasted due to tech companies becoming too big and the bureaucracy associated with it. It literally was the same with mechanical engineering companies before. Some people build a product in a small environment and once it becomes a global success a lot of additional people are needed to simply manage everything (also the technical side) and those people again need new processes with new internal stuff and so on. Additionally, a lot of "lost productivity" actually is not lost but being spent on scalability and usability. Simply looking at it from the number of features implemented per employee/time is pretty stupid when Facebook started out with a few thousand users that would post once per day compared to the millions of users and corporations that use Twitter nowadays with multiple messages and videos being posted daily. Apps and Websites have evolved a lot in terms of usability the majority of users are not interested anymore in spending hours on grey or black and white forum web pages that only support text with a terrible search function. Comparing the pure functionality of some mercedes a decade after it was invented to one now is basically the same. Both cars drive and get you somewhere but the comfort/usability of the two products are something completely different.
I mostly agree. But I'm a bit skeptical about the last point as well. Programmers may not be losing that much productivity at all. The charts that were shown in the video were labelled "number of employees" not "number of engineers." Maybe that includes help desk people and marketers and whatever. If someone has more convincing statistics, I would take a look.
this is Blow's whole vibe now. he programs all his games from scratch, even going so far as to write his own language. there is nothing in his games that remotely require that. he is the epitome of thinking he's the smartest one in the room
@@Raxfyr He's just what a painter is to digital art but for programming instead. Like, not everyone needs to be smashing prompts into a generative model or using digital painting software. Hand-painted artwork is still in demand and so are handcrafted games. It's not because you don't appreciate something that nobody else does. Get your head out of your ass a little
The big elephant in the room is that nowadays even if you program in assembly you still don't know what the CPU is doing with all the register aliasing, speculative execution, muop fusion, out-of-order execution etc
This was one of the contention points when a bunch of VHDL files on AMD cpus was leaked on the internet. The amount of people which understand hardware at component level like that wouldn't even have interest in those. They're already employed in well paid positions, as hardware designers are a very finite number of professionals these days. The dellusion that you know hardware at component level is very high amongst older programmers simply because they know lower level languages.
@@BrunodeSouzaLino Well, they also know old systems where that assumption, knowing what the machine did at the lowest level, held true. And we see the consequences of people losing this ability every day - that why we get stuff like electron or similarly bloated stuff that requires hundreds of megabytes in memory and assets to do the simplest thing possible. And the fact that people seem to be okay with that and just shrug it off like nothing.
@@lucemiserlohn There's two things Electron is good at: - Cross platform apps - Developing time Writing and maintaining cross platform native apps requires a lot of time and effort and it's an uphill battle with systems like Linux distros, where it is impossible to cater to every single case or set of apps each user has installed.
@@BrunodeSouzaLino I never said electron is not useful in certain cases; it's just bloated and painfully slow, both aspects which I personally do not want to deal with, neither on the developer side nor the user side.
there's a good usenix talk on this subject and OSDI, specifically that the "operating system" barely has any control over most of the hardware. most of the hardware communication happens via ad-hoc black-box firmwares written by who-knows-what company, with absolutely no cache coherence compared to the OS
It's a mistake to divide the "new features" at Facebook by the number of programmers at Facebook - most of what they are doing is not producing features that benefit users, but rather features that make Facebook more money, which are not visible to users (on purpose).
As well he did a mistake somehow thinking that first Unix made in 3 weeks was sign of productivity. The complexity of that Unix was so low, modern programmers would have probably finish it in 1 week I suspect.
I can't believe he didn't comment on the key thing Ken hintet at that allowed him to write the first unix kernel: Being left the F alone! Teams ruin productivity and creativity. The bigger teams, the less productivity. The tools we are handed are generally cripled with a hateful OS - and add to that all the crappy antivirus, "group policies", lack of admin rights, firewalls blocking you from downloading standard programmer tools, SDKs, etc. - all things that are actively slowing you down (or outright preventing you from doing your work which you then need to spend a lot of time circumventing.)
With teams you end up with a larger range of project scale that can be produced, but at the same time you end up giving each person on the team less stuff to do overall, ruining their productivity.
Yeah, everyone in management needs to read Deep Work. The amount of consecutive hours you can work with no distractions is the largest factor in someone's productivity, and most companies won't even let you ignore slack for 15 minutes.
Helps both of them are of the same generation (Terry Davis and Jonathan Blow), they are really productive people. I remember Terry Davis multiple rants regarding the English or Humanity people getting involved in programming and often doing things that are affront to computers and talked about how important it is to program the way the computer wants it to be done.
All good causes that he lists, but the biggest cause is the much higher demands on software today. I'm a third generation software developer and have talked with my dad and granddad about these things. Back in the day most programs were simple. Now you need a sophisticated and user-friendly GUI, Server/client or peer-2-peer communication over some protocol, complex databases, security, scalability, metric collection, statistics presentation, sophisticated rollback, integrations with all kinds of third-party systems. Plus a sea of technologies/frameworks/libraries/tools to get anything done because building your own XML-parser from scratch is stupid, but now you have to master all these things, where before you simply didn't need any sophisticated parser. Also as technology has allowed it more and more legislation and bureaucracy imposed by the state has effected the processes around software-development and even worse the software itself (GDPR for example). To all this add that users often demand more advanced features today (it was very basic what the software my dad made in the 80's had to do). Further more there are more bad software-developers. Many think being a developer is about writing code (as in you just need to know one or more programming languages in depth). It is correct that you write code, but think of it as articulating solutions so to speak (like when you make a mathematical proof and articulate it in a mathematical notation), but what is hard is recognizing the problems and conceive the solutions (like math). And you can't think of a solution and make somebody else code it because coding and crushing problems happen in tandem. Also, passing a complete solution to another human being would be the same as writing the code yourself.
I think that's half right. Developers just have more features to develop these days. But if all those features were useful and worthwhile then you could say that developer productivity was as high or higher than it used to be! The problem is in a lot of software, even a lot of games, but I'm more thinking of other software, only some of the modern features are useful and a lot of them are just pointless. Often people work hard at producing stuff that doesn't even need to exist, either by being misguided or not very bright, or deliberately, for money! Or spend time learning and using complex tools and processes where something simpler would work better.
The "sea of technologies/frameworks/libraries/tools to get anything done" is likely a main reason productivity is slow in the long-term. They give you a quick and dirty solution for prototypes but they make maintainability a pain and instead of learning what your code is actually doing, you're spending time learning arbitrary made-up pipelines and functions, trying to dig up what the black box libraries actually do under the hood - and when they're actually open source you find out that they're impossibly complex and will take a very long time to understand. But it doesn't have to be that way. It's because of modern software dogma of turning everything into modules, singular generalized objects and overreaching pipelines.
I don't agree that abstraction is to increase productivity at all, abstraction allows you to increase the complexity of a system. You can now package ideas and interface with them in a way that is human friendly meaning your brain can keep track of a lot of interacting components.
hiding complexity in for example interfaces is increasing productivity. I don`t think the number of components programmers keep track of has changed. It`s just that instead of pointers and a list addresses in ram we use containers, instead of writing 10th thousand lines of code we use for example network stacks. The level of components we use implements a very abstract function for us so within the limits of our brain we can design ever-enlarging systems.
statically linking opens another can of worms. You need to keep track of the security issues of the libraries you use. If you use a dynamically linked library and that library has a bug, you replace the library. If the programs are statically linked, you have to keep track of the libraries that each program uses, then you have to recompile all affected programs and redistribute them. Secondly the admin can't even tell anymore which programs are vulnerable and must rely on the diligence of the programmer. He has to check if any of the programs he wrote in the past uses a vulnerable library, recompile it and redistribute it.
Complexity is the source of most problems. I use templates in C++ not to introduce more complexity, but flexibility. You have to very careful about that abstraction most likely increases your distance from the actual problem u wanted to solve in the first place. And regarding speed the optimized version is most likely a very specific solution and not a general one.
I’m a senior engineer at a large tech company and most days all we do is circlejerk about patterns and refactoring code and never deliver anything. Everything is hilariously over complicated to the point where extremely basic stuff takes weeks to do.
Using the right abstractions should make basic things take less time, not more... no? I was in a similar situation (debating approaches eating up time), but I think "patterns/refactoring bad" is a red herring. For us, features took a long time in part because the code was fragile, full of cycles, and so inconsistent it might be considered abstract art. :P A more fundamental problem was that no one on the team (including me) had the expertise to say, "I've done something like this before, here's what I've seen go wrong ... I recommend approach X for reasons Y and Z," so that we could just get to it. That's how I learned expertise means a lot more than job titles; if no one has done the reps, we're always in uncharted territory. It also takes trust for that to work, and trust is earned. If we _didn't_ spend time on design, new code was fragile, hard to read, hard to debug, and exacerbated tech debt. If we _did_ spend time on design, it would take weeks to get to something that was _ok_, but it would have taken even longer to get to the 'right' (sustainable) solution. We didn't have any underlying goals/non-functional requirements to justify which refactorings were worth the time (e.g., "component X must be fault tolerant because it's the foundation for mission critical functionality Y", or "components A through F run
"I know what we need to do to get back to the days where we could write an operating system in 7 days!" says the guy who takes 7+ years to deliver each game.
That becomes a big hit.Often for the year. With sales in the tens of millions. for puzzle games. That are built by a small team not an immense AAA studio. As craft and not a template. That works when released rather than getting released expecting to bring it to a working state with updates and patching via the internet...
some whacky arguments being done here, large companies are eaten by bureaucracy, maintenance and very specially: communication among them. People and teams don't scale. Large companies and products try to avoid feature creep as it adds more maintenance, and collides with previous existing features and structures.
Today to be "full stack" you have to be 50% of a dev ops engineer in many an organization, especially in start ups and small-ish companies. That greatly interferes with software development productivity. Also too often today a large percentage of your job as software developer is to manage the use of up to a dozen SaaS packages that the CTO decided would "save effort and cost" each with their own peculiar merits, demerits, behavior as APIs and whatever the heck was in the heads of their devs. Banging on these to make it more or less satisfy business requirements dependably and with reasonable performance is a full-time job often. That leaves little time to write or think of writing the actual software that would most efficiently meet the business needs.
Fuck that shit honestly. Tryna save time and money by "saving software" is pure fucking folly. Especially with AI's these days. Everything is gonna need to be rewritten fully. Thats just unavoidable. Its less painfull though cus we got fucking AI. Its so braindead tryna modularize code and all, only for it to not even remotely matter cus of how much work u gotta put in to integrate it.
How is using one specific genius like Ken Thomson who worked at the most state of the art computer R&D lab at the time, a valid example to describe a whole generation of programmers. Its like saying that people in the 1900s were the smartest generation ever because Albert Einstein published his relativity theory at that decade. Makes no sense.
also it's ridiculous to imply that the world would be more productive if every programmer produced a new operating system every 3 weeks. we already have too many of the infernal things and this was something he himself was complaining about in this talk.
I think a lot of the "lost productivity" is actually being wasted due to tech companies becoming too big and the bureaucracy associated with it. It literally was the same with mechanical engineering companies before. Some people build a product in a small environment and once it becomes a global success a lot of additional people are needed to simply manage everything (also the technical side) and those people again need new processes with new internal stuff and so on. Additionally, a lot of "lost productivity" actually is not lost but being spent on scalability and usability. Simply looking at it from the number of features implemented per employee/time is pretty stupid when Facebook started out with a few thousand users that would post once per day compared to the millions of users and corporations that use Twitter nowadays with multiple messages and videos being posted daily. Apps and Websites have evolved a lot in terms of usability the majority of users are not interested anymore in spending hours on grey or black and white forum web pages that only support text with a terrible search function. Comparing the pure functionality of some Mercedes a decade after it was invented to one now is basically the same. Both cars drive and get you somewhere but the comfort/usability of the two products are something completely different.
the car thing is not really comparable i think, its just an old invention what got to business and drives humanity so badly, destroys earth ecosystem,, 4 wheels do the same over a century, the software things are really stupid to think about, i think its just a good hobby or a thing what you are exited about and goo to deal with it, its like music, should not be taken this complex anyway the real jobs are about survival and material things, building house, agriculture, but after all everything is being with a problem too, you cant except from the nature what you want exactly, somewhere, something, going to crash and you have to solve it somehow, or leave it and go on to the next thing
Sir do you know of the Longhouse? If not some there are some great articles, namely (google) "What is the Longhouse?" - for the more cultural take that goes hand in hand with what you said.
There is one thing we should keep in mind with the comparison of "number of developers employed" vs. "tangible developer efforts" at the big tech companies. They aren't necessarily hiring for productivity, I think. That hiring is, at least in part, driven by the need for infinite growth. They have to show that they're always growing to "make line go up." Later, they'll prune the developers they didn't actually need in the first place to show "fiscal responsibility" which will "make line go up." Still later, they'll overhire to "make line go up" again, etc.
The majority of software and features that is built is never used. Just requests by managers, then when the software or features is ready 3 months later, those managers are onto something else, the business case for the software doesn't make sense etc. There are about 2.7 million apps for android, about 1.000 apps per day. Yeah right.
It's quite common these days in all industries. Make new product with barely any need , product managers exaggerate what market wants or where market is headed just so you can sell something new. Re-inventing of the coke bottle but with software. Maybe its different at the top tech companies but a majority of software engineering seems to fall under this.
@@yalslaus the "top" tech companies are worse because they have a dozen teams all duplicating work because they've ended up not communicating with each other
Lisp is one of the most abstract languages around. At Symbolics, a company that sold Lisp-based hardware and software solutions, a VP once claimed their ambition was to make all but the largest of software projects achievable by a single developer. (Small teams would tackle the largest projects.) And it looked like they would be abke to do that. Small teams built Genera, the Lisp OS, their document management system, their expert system solver, and their suite of 2D/3D graphics tools that were the rival of anything produced by Pixar at the time. But large companies like large teams -- to a middle manager, a large team means a large budget and a large subordinate count they get to command -- and so as the capabilities of small teams grow, that extra productivity is taken away by the introduction of processes and methodologies designed to slow small teams down and necessitate large teams with large management infrastructures.
Incredible that so many things came out by using Lisp. I remember learning it in college and the recursion was hard to grasp, but the mental models created helped so much in the long run. One of the projects was creating a SQL like interpreter, using some helper library. I think our team finished it in a weekend.
Large companies like large teams because people quit. If one or two devs quit on a large project, it's okay. All knowledge is not lost, but if one or two devs quit from a small team, you lose a ton of knowledge that can not be regained without dedication.
I remember reading about a guy that when ever he created a user application he wrote an operating system to go along with it. Now, everything has too many parts just to have parts.
Here’s the thing. Imagine you want to make a fun puzzle game for a phone. Imagine how daunting it would be if step 1 was “write an operating system”. This is a wildly oversimplified understand of the systems we’re working on. Imagine you start a game on your phone with its own operating system. Your computer shuts down, and then boots you the game. Ok, let’s pause the game and check my email. Oh wait- hold on. I have to shut down my phone and boot up my email operating system. Abstraction is being able to use the efforts and expertise of others to enhance your work. Leave the OS engineering to the OS engineers. When you build a house, there are people that build the foundation, people that build the framing, people that put up the dry walls, and people that put on the roof. One person can do all of these things, but it’s much quicker for people that are specialized in specific things that do it every day to do that specific thing.
@@shableep It's quicker to delegate the labor to specialists but it results in a worse product. The best homes are built by one or a few people with a common vision and an understanding of ALL the systems involved. The reason software expires pretty much on arrival today is because of years and years of delegation compounded on top of each other. No single vision. Design by committee. Disposable garbage.
Basically how all Amiga games worked. It was very rare for a game to require Workbench. You just put the game disc in and it ran. So they basically all had their own min-os hardware routines written in assembly to do common tasks normally handled by an os these days.
The reason programmers are "afraid of pointers" is that raw pointer arithmetic leads to a massive amount of bugs, in turn leading to applications in production going haywire in de most unpredictable ways. Also, programmers to do low-level programming well they need lots of time to concentrate at the issues at hand ... which is pretty much an impossibility in this day and age of DevOps and Agile.
You don't even need to do pointer arithmetic yourself anymore. It's all handled by array indexing but people are still afraid of a fucking value holding a simple RAM address
The lot of time part echoes with me as a sysadmin - in my field productivity is also strongly linked to idle times. Since you're all-times split between daytoday and readiness to deal with outages and projects you need to spread buffers everywhere where your colleagues and you work on keeping their edge over the needs of the day to day. If you spend all time in agile standups, 1on1s or chasing your sprints fixing bugs you'll not be able to have that edge. Not be able to gain the distance from your current detail issue takes away the chance to coordinate with colleagues, and to idly spin ideas.
Sounds like a skill issue. But in all seriousness, this is just an easy copout. “Oh it’s too hard, oh I’ll make too many mistakes.” You’ll make mistakes regardless of if you use a low-level language or not, and especially if you don’t practice. By practicing using pointers, you’ll better understand how they work and eventually avoid making mistakes entirely. There’s also numerous memory management strategies, such as arenas, that simplify dynamic memory allocation quite a bit. Pointers are really not scary at all.
This is absurd. I can launch a cross platform website that works on multiple OS's including mobile devices and desktops, in a weekend, without having to build my own web browser.
3:43 The problem with example is that actually a lot of innovation within these big corporations gets shot down, And I say that because I know people who work or have worked at these kind of companies. Usually its either because of inner-corporate politics or because people think its not good enough, so saying that you don't see the productivity as an end user, doesn't mean devs are not being productive, they just cater to the whims of executives or seniors who don't appreciate that productivity or are too conservative and ultimately reject it.
No. The problem is people fantasize about big corporations wanting innovation or even needing innovation. They don't. Most organizations want you to competently make an API call between their accounting system and their time card system. They want you to glue disparate systems together so that their business functions in a cohesive way. Showing an example of a grade computer science pioneer genius writing an operating system at Bell Labs, whose only goal, with to spend as much money as fast as possible on R&D to average developer applying critical thought on solving a normal business problem, makes this man look like king of the shmucks. He is Shmuck Man, comparing of things that don't make sense. If you're being paid to say.... develop a time keeping system and instead built an OS, you'd be fired. Because you don't work in the whimsical word of Bell Labs where every developer has full autonomy and an endless budget that doesn't need justified and you don't even need a relevant business problem to solve. You can just "Do Stuff". In the real world, with real businesses not in 1980, businesses have narrowly defined problems they need solved. They need some one to randomly poop out an operating system for funzies.
I have had one year of total freedom in my job. I used it to set up everything the way I wanted it. TDD (actually writing tests first), CI/CD, continuous delivery (so no, not branching with git). Then I started hiring people and now I have 8 developers in my team. I taught them everything I know. And now I work in a company with 0 legacy code, perfectly readable tests. I've literally created valhalla for myself, but only because I had no co workers for a year and management that gave me enough trust to get this stuff done.
The API is fine. It’s that developers today struggle to use them. Once upon a time, APIs were a great way for software engineers to communicate and share with each other. Today, it’s a skill unto itself that requires a 2 week training course. WHY?!? Can’t we just read the documentation and try it out? If it doesn’t work for us, don’t use it. But in practice, we first invest in a multi-week training course, then claim certification on “the thing” and then we have to use “the thing” because we’ve invested so much sunk cost into it. That’s bananas!
@@thewiirocks Many corporate APIs have restrictive aspects to them that are not there to solve or aid in solving an engineering problem, but instead to give a business leverage. In cases where you have no other option but to go through an API you are at the mercy of the author of the API, and if it doesn't expose what you need you've now got an artificial problem on your hands that adds an unnecessary complication/hindrance to your work.
@@kleozane2781 Software companies fail for the same reasons that all companies fail. They are unable to provide a product desirable to the market at a price that is acceptable. Everything else is just noise and may or may not contribute to the failure of any one company.
The metaphor of measure productivity as just new features for productions (like Twitter and Facebook) is incorrect, or at least very incomplete. It is only one axis. You can take a product and change no external features, but go from 1000 daily users to millions. In order to support that you need a lot of different things to happen. There are ports to other platforms, internationalization and so on. I think there is also a point to be made about constant feature changes not actually being super desirable, especially as a product grows. The more users there are invested in how a product works, the more care it takes to change that product. The more tools and infrastructure you tend to have to support the evaluation and safe deployment of feature changes. And because of the scale of the user base, a small feature change that affects millions is probably "more productive" from a business perspective than many feature changes that affect only thousands of users.
Port to other platforms and internationalization isn't development per se, it isn't creation of something. Yes your house needs some fixes here and there but you don't need to rebuild the whole house every time with architects and engineers involved.
One reason for the lack of compatibility is that companies do it on purpose to compete with each other. They make their stuff unique so it's harder to move to someone else. They create a big support team and restrict third parties from providing support. Having full control over their product and not having to worry about conforming, they can optimize the user experience. Of course when considering the bigger picture, these reasons have immense drawbacks and do not benefit the industry as a whole or the users. "One who only cares about their own wellbeing can't possibly contribute to overall wellbeing."
I guess we shouldn't be surprised that software dev is following the same pattern as most other technological advancements. Something tends to be lost when something is gained technologically.
I think many people don't understand the "invisible feature" creep that comes with scaling. Example: a colleague of mine had to fix a bug coming from laptops moved around and connection getting lost as the new router was from another IP address. This was not a bug, as it was not a coding "defect" but it was also not a marketable feature. But it had to be understood, designed and tested just like any "sellable" features. The bigger the usage of a product the more of this has to be implemented.
The reason people were able to just create an OS in a few days is there were no standards. You complain that there's a shitton of different Shader languages? Besides vendor lock-in, which will likely attribute for some of it, the reason is that people went at it with your perspective. Why spend time trying to establish a standard, when we can just make stuff up ourselfes. The reason OS's complicate deploying Programms is because they allow one to use more than one program at the same time, and that requires coordination of shared resources. Yeah, if you'd write it from ground up PERHAPS (depending on what you're doing, probably not most of the time) you'd be quicker, but you won't be able to run that while also browsing on the same machine, and every program would have its own idiosyncraties that users would have to get used to.
While I get where he's coming from and agree with a fair bit of it, the company thing isn't entirely tech issues (not that the amount is zero). Software red tape in business seems to get worse by the year. I'm a fairly important developer at my work and yet there are more than a few weeks I spend more hours in meetings than I do getting to actually code. I wouldn't be surprised if FB/etc have the same problem.
This talk is hot garbage. Developer productivity is not approaching zero, and places where it is, it's usually the fault of crappy management, not the choice of a higher level language than assembly....
He is not blaming developers, at least I do not see it as blaming. He just states the fact that effectively most developers write almost no code. That's why I do not work as a programmer anymore - too many meetings, too much unproductive talking, no programming assignments. If you like coding, do not aim to be a corporate developer.
The bigger the company the more safeguards are in place which requires more input and review to release. I had a text change on a button take 4 days to get approved. Then when we switched our instance from a xlarge to a medium that took about 4 months of getting sign offs and approvals. All of which was 3 lines in terraform.
Having just an undergraduate degree in Electrical engineering I understand what he was talking about, but have three fundamental critiques. 1. He just states that the productivity of programmers is approaching zero and then doesn’t really justify the claim besides his twitter graph and a single anecdote from one bell labs engineer from the 60’s who probably wrote code that others had to meticulously parse through later for all the bugs. 2. Just because Twitter isn’t adding functions doesn’t mean its programmers are less productive. Every corporation has diminishing marginal returns as its workforce grows and software companies are no exception. Programmers could be working on a whole host of other things including maintenance, security, scaling, and testing. 3. All professions these days are highly specialized, yes it would be good for all programmers to know all languages and levels of abstraction, but it’s absurd to expect the majority of programmers to learn all of this if they only need to know the key technical literature and 2-3 languages really well to be productive at their job.
the real kicker is that even the OS doesn't have true hardware access. also the metric of "features per programmer" is the funniest metric i've heard of
Right!? Also, one of the main reasons civilization even exists in the first place is due to specialization of work. Now we're using that as a metric for the decline of civilization? I'm sorry. I'm not buying it after two measly graphs and one short video.
Small startup teams are incredibly productive. Giant teams with Project Managers, Scrum Masters, daily standups, etc are what slows things down. Bureaucratic red tape kills everything. Take Facebooks 5 top guys and let them work 100% autonomously. See what they can create. Nothing ever written in "older style" code could support 1 billion users.
I don't think that's the point, Jack. The point is baked into all of this advancement is also un-necessary complexity that reduces the ability and increases amount of work needed to achieve certain results. Try to keep up buddy.
Problem with the unix example is that those 3 things are not the only ones currently required for a popular OS. Things like visuals, design, security, sound, compatibility, updates, are completely necessary.
Necessary? For an operating system? Why exactly? The OS part of what you call the OS should be as small as humanly possible; everything else is a program running in userspace, and that includes the majority of what you listed there. The fallacy is that we think we need all that heavy ballast in the OS itself; and the better design angle is to get rid of it all and just treat it as ordinary software that can be run, but doesn't have to be run; that can be deployed, but doesn't have to be.
@@lucemiserlohn An operating system needs to be user friendly for the majority of people, the tech itself or concepts behind it don't matter for most people. They want something working straight out of the box with the most important features, that's why they pay for mac computers or windows and don't use linux. Companies adapt to what the average user or a niche wants, not the concepts. Also some things must be linked together, like user authority and reading or writing to files. If they were singular models/classes/functions, you would have a lot of boilerplate code, which could be ommited if the system had the dependancy of modules in mind, increasing the speed of the os, productivity of developers on that os and milestones per interval.
@@michaelatorn8380 That is not what an operating system is. That is a suite of software including an operating system hidden somewhere in there. Also, please define user friendly. I would argue that depends on which user you're asking whether it's friendly or not. Please also note that I did not say anything about users having to be technically minded. You can have an efficient, lean system and still not require the user to know the details about it. Also you assumption about Windows and Mac OS is wrong; people don't buy Windows or Mac OS on their own merits, but rather because it is what either comes pre-installed with their hardware, or because it's what they know from somewhere else or what other people tell them to use. For many, many people it boils down to either "does it run MS Office" or "does it run my games"; pretty much any other use case is irrelevant for the average Joe or Jane. The fact is that a lot of the technology that is en vogue today in the software world at large is problematic at best; we have those insanely fast machines and the software we use on them still manages to bring them down to a crawl. And that is the real shame in today's computing world.
You're completely missing the point, out of ego hurt. And it's understandable, since his examples and criticisms of 'modern programmers productivity' is ludacrisly stackhanovist. "If you can't invent an OS in 3 weeks then you're trash"... Yeah ok lol, F U.
@@michaelatorn8380 Visuals, design, security, " look pretty "; pretty its different for everyone, yes I do thank having nice UI, but sometimes it gets stupid, like an overdesign, If i bought windows I want to use windows, not mac, then why is W11 looking like mac? as I said, if i wanted a mac id buy a mac, get ur sh*t together and follow a philosophy
Some time ago, you could slap a few lines of code and publish it freely, even by newspaper and post. Now even before you publish your first MVP there has to be a licence, a strategy for GDPR, you need to make sure it is compliant with law and probably you also need to follow guidelines from publisher (like Google Play). On top, how much you integrate with other APIs like identity providers, payment processors etc the more chance is you will remove friction that lose users. So much boilerplate on top of value
The big tech companies of today are in a real mythical man month situation. Move any small piece of their humongous code base and something breaks. Even with thousands of the best engineers can't make much progress, adding more leads to diminishing returns. The best thing for the economy would be for big tech to go bust and release these engineers back in to circulation where they can create new, better software from scratch.
Nope. The most valuable service the Big Tech companies provide is keeping those n00bs away from the rest of us. FANG-employment-seeking is a disease, fortunately they are self-isolating.
Need to move to containerized microservices and CI/CD pipelines. Monolith codebases of intertwined dependencies are out, self-contained packages running in containers and communicating via messages is in. The development process has more overhead, storage/memory usage is much higher, and constant environment setup is a pain in the ass, but it solves the vastly bigger problem of fragile, interdependent monolith systems.
Distributed system is the correct Computer Science term. How many times Distributed system are rebranded ? RPC, CORBA, SOA, Microservices, REST, SOAP, EJB, DCOM .... By the time one generation of programmers figures problems and solutions and patterns, there is new generation of programmers that have to learn the same thing but with new name and the cycle goes on. Of-course programmers aren't productive because they spend time to learn new rebranded tools. And by the way, IBM is doing distributed systems since the 60's on OS and hardware level
his point is that we have created too much accidental complexity, that we are repeating ourselves solving that complexity with.. more accidental complexity
You can still statically link programs, and I would argue that this is a better model in any case. Flatpacks and other such solutions carry the dynamic link libraries with the program, but that is essentially what static linking does. Plus, most of the incompatibilities are from the .so/.dlls in any case. The underlying OS calls don't change (the actual calls to the kernel). Finally, using a TK instead of direct interface to the graphics subsystem solves most of the other issues. I advocate static linking myself, and I find it hard to believe people are yelling at me "you are wasting memory!". A megabyte is what now, a fraction of a cent in terms of ram, and you are worried about that. Statically linked programs load faster and are more stable than .so/.dll programs. Finally, I make all my programs live in a tree in user space. Scattering the program and files all over the operating system not only makes installation harder, but it makes the program parts harder to find and uninstall or update without an automated uninstaller/updater. For example, Eclipse is a very complex program, but it lives in a tree in a user directory. In this case installing consists of copying or unpacking a tar.gz file.
I statically link my programs. The only problem is if you make a program with a few gigs of assets and then does it really make sense to embeds all of those assets in your .exe ?
@@KANJICODER YES. That's why god made virtual memory. As in you stuck it in the binary, and its up to the OS as to how it wants to optimize loading its pages or not, or even shifting those pages to near line memory.
It's also about disk space.... If you've ever worked in a bigger projects, you'd know that a lot of libraries are linked dynamically for edge cases, as well as more support. In which case, if you link it statically the final executable can increase by tens of MiBs to even hundreds of MiBs of just bloat Now this might just be another problem in on itself where programmers dynamically link to everything they "might not need". If you've ever used gentoo linux, you'd know how much bloat most programs are, but just removing it is not really an option. The average user won't know how to configure the proper flags for their system, that's why those bloat are necessary.
@@scottfranco1962 for one program, now multiply that to how many programs do you have installed on your computer. I have around 22GB of shared objects/dynamic linked libraries. Sure a program won't use all of them at once, but that kind of redundancy is just a waste. You're just being that developer who refuses to optimize their code because it's 2022 and computers are fast anyway, just get a better computer.
It is inadequate to judge productivity by dividing changes to the software by number of employees. Are all the employees developers? No. Does productivity scale linearly with the amount of developers? No. Is it as easy to perfect a product compared to getting the first version of the ground? No. His point about shading languages essentially boils down to a lack of abstraction. If the differences between the different languages could be removed or ignored, the problem he mentions would disappear. He created very good games but this talk is a reminder of the fact that one should talk about what one really knows and refrain from talking about things one has an opinion on without truly being an expert.
As a non-programmer (teaching myself how to code right now), but a kid that always wanted to become one, I now realize after many years my younger self thinking "wait a second, WHY exactly is that?" while learning anything actually meant I was thinking like this. I literally couldn't agree anymore on this, this is incredible how accurate it feels to what I've always been thinking. Thank you for uploading this video.
He is talking nonsense to be honest. Ten years ago we couldn't have imagined what kind of sophisticated programs we're going to have, he is pointless. Programs are advancing, market is very demanding, competition is very high. You can't afford writing in assembly or C unless it's absolutely necessary. Point about Ken Thompson is utter bullshit, first Unix was pretty simple, and Thompson is one of the strongest engineers out there. Software development is in its current state is not at all caused by some accident. It's a very competitive market, no one can afford to build their own stuff from scratch (like web-frameworks, game engines, etc.), to learn memory management for 10 years. What Ken Thompson would do in 2 weeks modern Jr. Sw Engineer could import with one command or install in 10 seconds (talking about productivity). He just pulled it out of his ass.
I see it form this simple perspective - we have abstractions only because of advancements in different areas too frequently so releasing new versions for that would be very time consuming, for example imagine you have a software that you wrote in low level programming language and then one of the OS you support introduces a new algorithm to their API, now instead trying to recreate that algorithm in your software you may want to take advantage of that feature, so you introduce different OS SDK s into your system, there you start creating abstractions because without it, you need to create dozens of parallel versions of your software
You say you don't like managed languages but point out one of the main problems that led to their creation. If you want to decrease your chances of having a memory leak, you use a garbage collector or something like Rust (maybe C++ with clang-tidy, but that's not as comprehensive).
Memory arenas and circular buffers are common solutions to memory management problems - you only do very few permanent memory allocations for each subsystem and then never worry about it again. If you program specialized batched functionality instead of generalized and atomized object-lifecycles then it gets a lot easier - because you can just step through the code and literally see what's going on without keeping track of complex blackbox states (and their possible synchronization requirements). Similarly, multithreading is often deemed very complex and costly but if you see "critical sections" as an inherent design mistake, then it becomes virtually a non-issue to manage.
It doesn't matter what language you program in anymore. On any modern platform, all you're really doing is feeding parameters to application programming API's, which you're orchestrating to obtain access to operating system services that implement the functions you need to automate. You're basically just embellishing the platform''s built-in functionality with your own branded UI, combined with whatever external content you're bringing to the table.
Never disagreed so much about a talk. We ask a lot today to software. Be secure, be stable, no OS corruption, be responsive, be compatible, and have features they didn't have back then. The lack of norm for shaders: we have few GPU vendors, we had a lot years ago. You had to write code for different GPUs, for different architectures. What makes a programmer less productive: open spaces, large teams, constant reporting, micromanagement, having to negociate for every demand to security departments, and so on. Sure it was easier before, when lack of security was not a issue...
pretty sure we can blame over-use of OOP for this, they end up working around/fighting with the structure that they themselves have created, to abstract away things that they didn't even need to.
Well I guess at a time when Unix was written programming was something nerdy for scientists. And the market with all this managers/business analytics was not affecting the programmers work flow that much. Now you need make people that far from technology believe that you are actually working.
I think a large portion of the issues come from how tightly coupled certain software is. Good luck working with k8s without docker or podman. Did you just want a simple drop down in a front end? Well you're going to need 65 packages and one of them IT won't let you use because it has a vulnerability (hyperbole, but only just).
The examples given near the start of the talk don’t match up with the explanations given at the end. One of his implicit assumptions in his first example was that Twitter hired a lot of engineers to program new features. From that assumption he concludes that the engineers are not being productive because of the ratio between engineers and new features. We now know that Twitter hired engineers to give the appearance of growth and increase the perceived financial value of the company, and not necessarily to build and push a bunch of new features. The issue in his practical example does not lie with the tools being used to build software, but rather with free market capitalism. The rest of the talk is an argument that the complexity growth in software tools is not worth it overall. The issue with that argument is that he isn’t explicit in the metric he is using to measure what makes software “good”. By the end of it, I was pretty sure that his issue was with how the monetization of software has impacted how we create software. I’m not sure that he realizes that though.
I write code at least 2x times smaller than others all the time. A team of five programmers produced about 80 percent of the excess code. Imagine if you have 100 of them…
It's the kind of people that write code like: if (x=0) then return (12); else if (x=13) then return (1); else return (x); Instead of: return (((x+11)%12)+1);
From the point of view of just coding that could make some sense, but business related if making an oversimplification of ‘productivity’ , just because you don’t see many features in the apps you mentioned does not mean it takes more time to program, but features go through a life cycle
*Just create a small simple UI hello world program in WPF/WCF ie c# and use spy++ to find out the multiple layers of HWND it has created unnecessarily.*
This is going to blow your mind, but: the world doesn't need any more hello world programs. No, really! It doesn't. The world needs, and people are prepared to pay money for, richer and more complex applications. And that's where UI frameworks come in, for which the stuff you call "unnecessary" has a purpose.
@@BittermanAndy Funny you say that while most big corpos are literally cutting features out of their apps on the daily. Everything is getting simpler, not more complex. RU-vid used to have full-blown customization for channels and now every single channel looks the same. MySpace allowed music and custom HTML/CSS for profiles, now Facebook, Instagram, TikTok, Reddit, and Twitter are all generic profile pages. You don't know what you're talking about. Like, at all
I think one of main reasons for this crippling exponential overcomplication is that software is nowadays used in exponentially more interlocked human society senarios. Its probably same with the law. We just live in a complicated interdependent society.
Part of the problem is the low hanging fruit is gone, aside from unnecessary added complexity. It gets harder to make changes or rather add functionality with increasingly more complex systems, requirements, security and underlying architecture, or the lack thereof. Lack of documentation and descriptions of what things were intended to do etc
Hear hear! We let the frog stay in the pot too long. Someone once calculated that to get a pixel on the screen in Java, the process goes through 17 layers of abstraction. We have to "install" programs instead of copying them over. I loved programming in assembly as well but all that went away.
The market made a choice. You can still program in assembly if you want. Grab an Arduino starter kit. No abstraction at all if that's what you want. And the target market for the software you write is zero. Have fun.
I watched the full original video a while back and I had significantly less experience than I have now. I have to say that this ignores the fact that a lot of systems are now distributed to provide high availability, and to scale with the massive amount of users and fluctuation of traffic - and those are legitimately tedious and difficult problems to deal with. There's a lot of infrastructure configuration (for people, hardware, and software all the like). The largest tech companies are successful at managing and deploying software at scales you don't deal with when you create single-player video games. I'm not denigrating Blow's games in any way, I think Jonathan is the best video game designer ever, bar none and the stuff he's made are masterpieces, but the arguments provided on this point are a little bit flimsy. The other point he brought up in this talk about 'losing' technology is something really scary that I can't help but feel is totally accurate.
A lot of it doesn't need to be built this way to begin with, however. We have entered an age of centralization. Decentralised alternatives could be safer (can't access millions of accounts with one hack), would be more efficient and less prone to walled garden problems arising.
Although Unix OS is widely used and is highly important, the initial release was not that complicated compared to modern standards of programming. Plenty of CS undergrads build OS’s or compilers for their own language. The number of cases you have to cover for these are actually quite small, even though they are so pervasive.
The number of software companies has drastically increased over the last twenty years and so have their product/service offer. In this sea of competition, each brand is chipping in that extra feature to win new clients and sell their software, thus increasing the complexity of the original idea and influencing the developer job market with fads and trends over the need for new requirements. There are companies out there rejecting front-end developers without SASS experience because on their resumes they see only CSS. Food for thought.
right? you don't need an installer. on windows, all an installer does is add registry keys and copy some binaries and maybe create a service. on linux, all an installer does is add some config files and copy some binaries and maybe create an init script... you get the idea
Exactly all installers rly do is copy the file to a certain directory and add like a shortcut. Even with something like a flatpak i can just take the binary move em some place else an run it just fine
It's true for single-host solutions in monolith architecture, but once you deal with microservices that have dependencies and which are distributed among multiple servers, then you absolutely need something that is essentially an installer to configure all services to work with each other and upgrade everything without turning installing into full time job.
@@Gnidel Instead of microservices, we should make nanoservices for our microservices. Then I heard it'll be even more flexible and I'm sure it won't have any bad consequences
The objective of abstraction is to get high level functionality for cheaper costs. You can take cheap novice dev talent and have them code in Java and you can still deliver and it’s significantly easier to do that at an enterprise level
Twitter and Facebook are awful examples, most of the staff are not programmers at all. At Twitter, Musk fired almost 80% because they were censoring all day. There is also the large company's disease of being inefficient for wasting too much time on meetings and increased bureaucracy. As an indie, Jon Blow does not have these problems. However he may encounter performance issues that require very specific knowledge and experience. Even the large companies can have difficulty finding it.
I get a strong sense of similarity between his description of the bureaucracy of software and the bureaucracy of politics. A lot going on but most of it is ceremony.
I’ve noticed that most of the apps, websites, computer games, software that I use at work are very buggy and badly designed. They crash and lag and slow down even when the program itself is not very complicated. I think that the talent pool in the industry is diluted and that new employees are getting lost on the code previously made by former employees. RU-vid is the one of the worst offenders. It seemed that they’ve been messing with the code too much and are unable to recognize when it’s time to stop updating and revising the programs. It has bugs that weren’t there a few years ago. The algorithm has changed and doesn’t perform as it used to. Certain UX features now cause problems with playback. Maybe someone can help me understand why this seems to be the case or if i’m not taking into account important variables on my statement. I’m an amateur programmer so i’m quite ignorant.
As your application grows, it gets significantly harder to maintain. Changing one small piece can have large effects on other parts of the system. RU-vid has actually added a lot of features, including 'Download', 'Clip' and 'Super Thanks' etc. for the end-user, and a lot of analytics tools for creators. They've also done a lot of experimental changes to layouts, removed the dislike counter display etc. Some features might seem extremely simple but implementing them at the tremendous scale RU-vid does has many other issues you need to deal with. If you think the 'like' button is just about storing a 'like' counter in the database, you're probably right, but it's not that simple. Every 'like' is going to be associated to the person who did the 'like' (you can view your list of liked videos), so it's possible they store the 'like' as an association between your profile and the video that you like... but if it's stored that way, how are you going to display the counter now? It's not just querying one value from the database, you now need to aggregate all the likes from all users every time the video page is visited (so that you can display that updated value to the visitor) - imagine having to search through all users' profiles, looking at whether they liked that video, and counting that as '1 like', adding all of that up and displaying that as a number on the video page every single time someone visits it. This is obviously incredibly inefficient though you get the most up to date like-count this way. RU-vid almost certainly stores and reads the 'like' counter directly for the page, and then periodically updates it based on aggregating the likes across users; this is why you don't see the like/view counter updating in real-time or changing every single time you refresh the page. Now what about providing notifications to the uploader about a video that was liked? How do you efficiently store data that's going to be queried by hundreds of millions of users everyday? Do you need a distributed in-memory cache to handle the like counter and write that value to the database much less frequently? What about when a video gets liked and unliked and liked again? What about the recommender systems that interact with a video's like count? Should a creator wake up to see 10,000-100,000 notifications on a video for all the likes and comments? Can you imagine maintaining a site that deals with 1 billion hours of video uploaded every day? There's a lot of engineering going on that you can't see to enable the site to run so smoothly at this scale.
RU-vid’s backend is solid. Frontend has always been a trade off between usability and aesthetics. RU-vid at this point seems to prefer aesthetics. Software quality hasn’t really gotten worse. People today don’t remember the reams of awful corporate GUIs and win32 apps built in the 90s and early 2000s. Apps that had the possibility to bluescreen your computer if they crashed (as opposed to just crashing the browser). I think people mostly remember the games. Specifically, the games that were good. Plenty of bad, buggy, and slow games were made in that era too.
Nobody is doing lego bricks from scratch anymore but you must use predefined legos and sometimes well the legos cannot build what you want and you must do crazy construction to get the job done. Nasty.
One of the reasons the OS removes capabilities is deliberate: it's to predict intellectual property by drastically limiting portability. That, it seems to me, is the entire purpose of the "registry" that MS introduced in W95. That's not the only example, it just happens to be historically the first one I noticed at the time of introduction.
Idk what is this guy talking about, I wrote a lot of code in most of the companies I was employed, they constantly require new functionality. It was not rocket science, but new features are being added constantly, actually helping operators do whatever they do a little bit faster and you could see it in profit reports. Sometimes I feel unproductive but other times I actually felt I gave them too much value for my compensation, especially during my junior times. Maybe this is facebook-twitter related problem or just american?
I agree with him at some degree : if i see how the web technologies evolve and how fast some technologies are then thrown away, he is right ! All the Javascript code that is developed to then be rewritten with some other frameworks...
Depends on how you define productivity. I feel a lot more productive now than 20+ years ago. But I need to consider more things now for any user equirement. You can't base productivity on the time it takes for you to say "job done". It is like comparing the project time for a tiny house and a skyscraper.
I don't understand what this kind of argumentation is meant to achieve. We used to pick potatoes by hand, now we use tractors. We used to calculate with our hands, now we have calculators. All this progress is positive, in math, the education has more depth and students today can tackle new problems faster, problems that carve the nature of math at a much deeper level. This is just some old as dust boomer doomsaying by a guy who has somewhat authority because he made a couple successful games. This perspective is that code itself or the practice of coding (cough, excuse me, programming) is some kind of virtue, no matter what that code achieves, while the more practical view is, what the code does, what kind of value does it add to our society. Also, he seems to measure productivity by the amount of lines written, while the real measure of productivity is tightly coupled to the end result. If we held a competition to make a game in one day, one guy would start from scratch and one would use an engine, the engine guy would win by lightyears.
Productivity going down as projects grow bigger is nothing surprising. It's the same in every single industry. Decreasing yields. Once low hanging fruits have been picked, once you factor in the cost of communication, scaling, of course productivity is going to decrease. Now there are a some legitimate specific points mentioned in the videos and in the comments of things that could be improved. But for the most part, decreased productivity is expected and no big deal. It's literally economics 101. marginal cost increase and marginal gain decrease and companies are rationally expected to keep producing until their marginal profit zeroes out. It may be depressing for individual, and I for one much prefer green fields, but it's being extremely naive to expect the same level of visible innovation (new features) now as there were from an era with few and exceptional dev and most of the tools were yet to be made.
I think he's oversimplifying the "copy a program" problem. The average program is not just interacting with the CPU state. You need to also consider all the peripherals that are connected to the CPU. If the application wants to get user input nowadays, there needs to be some software or firmware component that can read USB. If it wants to communicate with a graphics card, that's PCIe or maybe on-chip, and if it needs support for multiple GPU models, it needs multiple drivers. What about persistent storage? To accommodate everybody, it needs to support SATA/SAS/PCIe/NVMe/USB, and it may need to use multiple at the same time to build the path to a device. When you see all those complexities outside the CPU, then an OS starts to make sense. Instead of an application bundling every driver that any user might need, the OS can provide that for them and even provide an abstract interface, so the application doesn't have to know the specific details. Fundamentally, I don't think this is a problem. We come up with new hardware standards and OS ABIs because they innovate and improve on the previous ones. Computer hardware is still moving forward and creating new demands for software support, and it makes a lot more sense for OSes to handle it instead of every application developer. Yeah it's annoying for software developers and distributors that OSes are incompatible with each other, but it's still very much manageable.
